Your API's hidden agent experience: what bots actually see when they visit your docs

Two versions of your website exist

There's the version humans see: polished, interactive, with animated code examples, dropdown language selectors, and a "Try it" button that opens an embedded API playground.

Then there's the version AI agents see: a wall of <div id="root"></div>, a bundle of minified JavaScript, and a 403 Forbidden if they show up with the wrong user agent string.

These two experiences are diverging fast. We fetched 10 popular API documentation sites the way an AI agent would — raw HTTP requests, no browser rendering, no JavaScript execution — and compared what we found to what a human would see in Chrome.

The results explain why 60% of AI agent integrations break at the documentation-reading stage.

The experiment

We picked 10 well-known developer platforms across different categories:

Company	Category	Agent-Readiness Score
Stripe	Payments	68
Twilio	Communications	68
GitHub	Developer tools	49
OpenAI	AI/ML	67
Anthropic	AI/ML	67
Slack	Collaboration	75
Vercel	Cloud/Hosting	68
Supabase	Backend-as-a-Service	75
Linear	Project management	68
Notion	Productivity	67

For each site, we ran five checks from an agent's perspective:

1. Fetch /docs with a standard user agent and with a GPTBot user agent
2. Read robots.txt for AI crawler restrictions
3. Check llms.txt and llms-full.txt for agent-optimized content
4. Inspect response headers for caching, content type, and bot signals
5. Compare raw HTML to rendered content to measure the SPA gap

Finding 1: The SPA rendering gap is massive

When an AI agent fetches a documentation page, it gets raw HTML. It doesn't run JavaScript. It doesn't wait for React to hydrate. It doesn't load web fonts, render syntax highlighting, or populate sidebar navigation from an API call.

Here's what we actually received when fetching /docs endpoints:

Domain	HTTP Status	HTML Size	Usable Content?
stripe.com/docs	301 → docs.stripe.com	242KB	✅ Server-rendered
twilio.com/docs	301 → www.twilio.com/docs	134 bytes	❌ Redirect only
github.com/docs	200	242KB	✅ Server-rendered
openai.com/docs	403	6.9KB	❌ Blocked
anthropic.com/docs	302 → www.anthropic.com/docs	143 bytes	⚠️ Redirect chain
slack.com/docs	Redirects	Variable	⚠️ Requires follow
vercel.com/docs	200	~180KB	✅ Next.js SSR
supabase.com/docs	200	~150KB	✅ Server-rendered
linear.app/docs	200	~120KB	✅ Server-rendered
notion.so/docs	Redirects	Variable	⚠️ API docs on separate domain

Three patterns emerge:

Server-rendered docs (Stripe, GitHub, Vercel, Supabase, Linear): The HTML contains actual documentation content. An agent making a GET request receives readable text, code examples, and navigation structure. These sites work.

Client-rendered SPA shells: The HTML is a thin shell with JavaScript bundles. A human in Chrome sees a beautiful docs site. An agent sees <div id="__next"></div> and 400KB of bundled JavaScript. The actual content is invisible without a browser runtime.

Blocked or redirect-heavy (OpenAI): The server returns a 403 before the agent even sees HTML. This is the worst case — the documentation literally doesn't exist from an automated perspective.

Why this matters

The most common way AI agents learn to use an API is by reading documentation. If your docs are a client-rendered SPA, agents need to:

1. Launch a headless browser
2. Navigate to the page
3. Wait for JavaScript execution
4. Wait for API calls to complete
5. Parse the rendered DOM

That's expensive, slow, and fragile. Compare it to curl https://docs.stripe.com/payments → immediate, parseable HTML. The cost difference is 100x.

Finding 2: llms.txt adoption is uneven but growing

The llms.txt standard (proposed by Jeremy Howard in September 2024) provides a machine-readable map of a website's most important content. It's like a robots.txt for content discovery rather than access control.

Here's what we found:

Domain	llms.txt	llms-full.txt	Quality
stripe.com	✅ 200	❌ 404	⭐⭐⭐⭐⭐ Exceptional
twilio.com	↪️ 301	↪️ 301	⭐⭐⭐ (redirects to docs)
github.com	❌ 406	❌ 404	Not supported
openai.com	❌ 403	❌ 403	Blocked
anthropic.com	↪️ 302	↪️ 302	Redirect (docs.anthropic.com)
slack.com	✅ 200	❌ 404	⭐⭐⭐⭐ Creative and thorough
vercel.com	✅ 200	↪️ 307	⭐⭐⭐⭐ Comprehensive
supabase.com	✅ 200	❌ 404	⭐⭐⭐⭐ Well-structured by language
linear.app	✅ 200	✅ 200	⭐⭐⭐⭐ Both files, complete
notion.so	↪️ 301	↪️ 301	Redirect chain

6 out of 10 have some form of llms.txt, though implementation quality varies wildly.

The good: Stripe's llms.txt

Stripe's llms.txt is the gold standard. It's structured like a table of contents for an AI agent:

# Stripe

> Stripe is a technology company that provides financial 
> infrastructure for businesses.

## Payments
- [Stripe Payments](https://stripe.com/payments): Accept payments
  online and in person globally...
- [Payment methods](https://stripe.com/payments/payment-methods):
  Acquire more customers and improve conversion...
- [Stripe Payments documentation](https://docs.stripe.com/payments.md):
  Find a guide to integrate Stripe's payments APIs.

## Connect
- [Stripe Connect](https://stripe.com/connect): The world's most
  successful platforms and marketplaces...

Every link is annotated with a description. Content is organized by product area. Documentation links point to .md versions of pages that return plain markdown instead of HTML. This is what "agent-first developer experience" looks like.

The unexpected: Slack's llms.txt

Slack took a different approach — their llms.txt opens with ASCII art of the Slack logo and a genuinely entertaining introduction:

> Welcome, humans and bots alike, to Slack's official LLMs.txt file!
> Of all the gin joints and internet rabbit holes you might have 
> fallen into, you've found yourself here instead.

Beyond the personality, the content is substantive. It's designed to shape how AI assistants describe Slack when users ask about it. It's marketing to robots — and it's smart.

The organized: Supabase's llms.txt

Supabase splits their agent-readable content by language/SDK:

# Supabase Docs

- [Supabase Guides](https://supabase.com/llms/guides.txt)
- [Supabase Reference (JavaScript)](https://supabase.com/llms/js.txt)
- [Supabase Reference (Dart)](https://supabase.com/llms/dart.txt)
- [Supabase Reference (Swift)](https://supabase.com/llms/swift.txt)
- [Supabase Reference (Python)](https://supabase.com/llms/python.txt)

This is brilliant for agents. Instead of one massive file, there are targeted references per language. An agent building a Python integration fetches the Python reference — not all 6 SDKs.

The complete: Linear's llms.txt + llms-full.txt

Linear is one of the few sites that publishes both llms.txt (summary) and llms-full.txt (full content). The summary provides navigation. The full file provides complete documentation in a single fetchable document. This is the ideal pattern — agents can start with the summary to find what they need, or ingest the full file for comprehensive context.

The missing: GitHub and OpenAI

GitHub returns a 406 (Not Acceptable) for /llms.txt — presumably their routing rejects the request before it reaches a handler. OpenAI blocks it entirely with a 403. The irony of the company behind GPT blocking the GPTBot-friendly content standard is... noted.

Finding 3: robots.txt tells you almost nothing anymore

All 10 sites in our sample allow AI crawlers. No blocks on GPTBot, ClaudeBot, or CCBot. This matches our broader benchmark data: 92% of SaaS companies don't block AI crawlers in robots.txt.

The real access control has moved to the application layer — 403 responses, JavaScript rendering requirements, CAPTCHAs, and WAF rules that robots.txt says nothing about. An agent can be "allowed" by robots.txt and still get a blank page because the content requires client-side JavaScript to render.

robots.txt remains relevant for crawlers that respect it (Google, Bing, AI training crawlers), but for AI agents that make real-time API calls, it's rarely the bottleneck.

Finding 4: The agent-friendliness spectrum

Based on our testing, these 10 sites fall into four tiers of agent experience:

Tier 1: Agent-native (Stripe, Supabase)

• Server-rendered documentation that returns real content on GET
• Published llms.txt with structured, annotated links
• Markdown-formatted documentation endpoints (.md URLs)
• Clear API reference with code examples in plain text
• Agent-readiness score: 68-75

These sites treat AI agents as a first-class audience. Documentation is written to be readable by both humans and machines.

Tier 2: Agent-accessible (Vercel, Linear, Slack, GitHub)

• Server-rendered docs (content available without JS execution)
• llms.txt published (Vercel, Linear, Slack) or docs accessible via standard paths (GitHub)
• Good API documentation with examples
• Some friction points (redirect chains, authentication walls for certain content)
• Agent-readiness score: 49-75

These sites work for agents, but the experience hasn't been specifically optimized for them. An agent can read the docs but may need to handle redirects, follow chains, or work around occasional JavaScript-heavy pages.

Tier 3: Agent-possible (Notion, Anthropic, Twilio)

• Documentation requires following redirect chains
• Docs may live on separate subdomains (developers.notion.com, docs.anthropic.com, www.twilio.com/docs)
• llms.txt redirects rather than serving directly
• Content is there but discovering it requires multiple HTTP requests
• Agent-readiness score: 67-68

An agent can use these sites, but the path from "find the docs" to "read the docs" involves detective work. Multiple redirects, subdomain discovery, and sometimes client-rendered content at the end.

Tier 4: Agent-hostile (OpenAI /docs)

• Returns 403 on programmatic access
• llms.txt blocked
• Documentation requires browser-based access
• Agent-readiness score: 67 (still decent overall due to API quality, but documentation access is poor)

Paradoxically, OpenAI — the company building the AI agents that need to read documentation — blocks programmatic access to its own docs. The API itself is excellent and well-documented for humans, but the docs themselves are behind a wall that agents can't easily cross.

(Note: OpenAI's platform.openai.com has excellent API documentation that's more accessible. The 403 applies to openai.com/docs specifically.)

What an agent-optimized docs site looks like

Based on the best practices we observed, here's the blueprint for documentation that works for both humans and AI agents:

1. Server-side rendering for all content pages

// Next.js example: ensure docs pages are server-rendered
// pages/docs/[...slug].tsx
export async function getStaticProps({ params }) {
  const doc = await getDocContent(params.slug);
  return {
    props: { doc },
    revalidate: 3600, // ISR: regenerate every hour
  };
}

If your docs framework is pure client-side (Docusaurus SPA mode, GitBook, etc.), enable server-side rendering or static export. The goal: a curl to any docs page should return readable HTML.

2. Publish llms.txt and llms-full.txt

# Your Product Name

> One-line description of what your product does.

## Getting Started
- [Quickstart](https://docs.yourapp.com/quickstart.md): 
  Get up and running in 5 minutes.
- [Authentication](https://docs.yourapp.com/auth.md): 
  How to authenticate API requests.

## API Reference
- [REST API](https://docs.yourapp.com/api/rest.md): 
  Full REST API reference with examples.
- [Webhooks](https://docs.yourapp.com/api/webhooks.md): 
  Configure and handle webhook events.

Key patterns from top implementations:

• Annotate every link with a brief description (1-2 sentences)
• Point to markdown versions of pages when possible (.md URLs)
• Organize by user journey (getting started → core concepts → API reference)
• Keep the summary under 50KB — link to llms-full.txt for complete content
• Update regularly — stale llms.txt is worse than no llms.txt

3. Serve markdown alongside HTML

Stripe's pattern of serving .md versions of documentation pages is the most agent-friendly approach we've seen:

GET https://docs.stripe.com/payments         → HTML (for humans)
GET https://docs.stripe.com/payments.md      → Markdown (for agents)

If you use a docs framework like Mintlify, Nextra, or Docusaurus, this can often be enabled with configuration. For custom docs sites:

// Middleware: serve markdown version when .md extension is requested
app.get('/docs/:slug.md', async (req, res) => {
  const content = await getMarkdownContent(req.params.slug);
  res.type('text/markdown').send(content);
});

4. Include machine-readable API specs

Beyond documentation, publish your API specification in a discoverable location:

/.well-known/openapi.json     → OpenAPI 3.x spec
/.well-known/openapi.yaml     → OpenAPI 3.x spec (YAML)
/.well-known/ai-plugin.json   → ChatGPT/agent plugin manifest

An agent can fetch the OpenAPI spec once and know every endpoint, parameter, response format, and authentication requirement without parsing documentation.

5. Don't block programmatic access to public docs

If your documentation is public (not behind a login), it should be accessible to any HTTP client:

# nginx: ensure docs return content for all user agents
location /docs {
    # Don't require specific user agents
    # Don't serve CAPTCHAs on documentation pages
    # Don't rate-limit reads below 100 req/min
    proxy_pass http://docs_backend;
}

Rate limiting is fine — you don't want a bot downloading your entire docs site every minute. But a 403 on public documentation is a signal that your site isn't ready for the agent era.

The business case

This isn't just about developer experience. It's about discoverability and adoption.

When an AI agent is choosing which payment provider to integrate with, it evaluates options by reading documentation. If it can fetch Stripe's docs in plain markdown and gets a 403 from yours, Stripe wins by default. The agent doesn't file a support ticket — it picks the path of least resistance.

In our benchmark of 844 SaaS companies:

• Sites with llms.txt score an average of 14 points higher
• Sites with server-rendered docs score 8 points higher than SPA-only sites
• The top 5% of agent-ready sites all have some form of machine-readable documentation

The hidden agent experience isn't hidden by choice. It's hidden because most companies never thought to look at their website from an agent's perspective. Now you have.

---

Check your own agent experience at anon-dev.com/benchmark. The score breakdown shows exactly where agents struggle with your site.