Designing APIs that agents can discover: OpenAPI, MCP, and the new standard stack

The discovery problem

An AI agent needs to use your API. Here's what happens today:

1. The agent searches the web for your documentation
2. It finds your marketing site, navigates to /docs
3. The docs are a client-rendered SPA — the agent sees <div id="root"></div>
4. It tries common paths: /api, /developers, /api-reference — some redirect, some 404
5. It finds something resembling API documentation, but there's no structured spec
6. It reads human-language descriptions and guesses at endpoint URLs, parameter names, and authentication requirements
7. It makes its first real API call. It's wrong. It retries with different parameters. Still wrong.
8. After 15 attempts, it gives up and tries a competitor.

This process takes minutes for a human developer (who can read HTML, interpret screenshots, and follow hyperlinks intuitively). For an agent, it's an unsolved maze.

The fix isn't better AI. It's better discoverability. Agents need machine-readable paths from "I know this product exists" to "I know every endpoint, parameter, and authentication requirement."

In 2025-2026, a standard stack for API discoverability is coalescing. It looks like this:

┌──────────────────────────────────────────────────┐
│  Layer 5: MCP (Model Context Protocol)           │
│  → Dynamic tool discovery + invocation for LLMs  │
├──────────────────────────────────────────────────┤
│  Layer 4: llms.txt / llms-full.txt               │
│  → Content map optimized for language models     │
├──────────────────────────────────────────────────┤
│  Layer 3: OpenAPI Specification (3.0 / 3.1)      │
│  → Complete machine-readable API definition      │
├──────────────────────────────────────────────────┤
│  Layer 2: RFC 9727 api-catalog                   │
│  → Well-known URI for API catalog discovery      │
├──────────────────────────────────────────────────┤
│  Layer 1: robots.txt + sitemap.xml               │
│  → Basic crawl permissions and page listing      │
└──────────────────────────────────────────────────┘

Each layer builds on the previous one. Let's go from bottom to top.

---

Layer 1: robots.txt and sitemap.xml (the foundation)

You already have these. The question is whether they help agents.

robots.txt tells crawlers what they can access. For agent discoverability, the important thing is not blocking your documentation or API reference pages:

# robots.txt — agent-friendly version
User-agent: *
Allow: /docs
Allow: /api
Allow: /developers
Allow: /llms.txt
Allow: /llms-full.txt
Allow: /.well-known/

Sitemap: https://example.com/sitemap.xml

sitemap.xml lists your pages. Including your API documentation pages in the sitemap helps agents (and search engines) find them:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/docs/api-reference</loc>
    <lastmod>2026-02-15</lastmod>
    <priority>0.9</priority>
  </url>
  <url>
    <loc>https://example.com/docs/authentication</loc>
    <lastmod>2026-02-10</lastmod>
    <priority>0.8</priority>
  </url>
  <!-- Link to OpenAPI spec directly -->
  <url>
    <loc>https://example.com/openapi.json</loc>
    <lastmod>2026-02-20</lastmod>
    <priority>1.0</priority>
  </url>
</urlset>

Most SaaS companies already do this reasonably well. In our benchmark of 844 companies, 92% don't block AI crawlers in robots.txt. The foundation is there — what's missing is everything above it.

---

Layer 2: RFC 9727 api-catalog (the new standard)

Published by the IETF in June 2025, RFC 9727 defines a standard for API catalog discovery using well-known URIs. It's the robots.txt equivalent for APIs — a single, predictable location where an agent can find every API a domain publishes.

How it works

An agent fetches:

GET https://example.com/.well-known/api-catalog

And receives a JSON linkset:

{
  "linkset": [
    {
      "anchor": "https://api.example.com/v2/",
      "item": [
        {
          "href": "https://api.example.com/v2/openapi.json",
          "type": "application/vnd.oai.openapi+json"
        }
      ],
      "service-desc": [
        {
          "href": "https://api.example.com/v2/openapi.json",
          "type": "application/vnd.oai.openapi+json"
        }
      ],
      "service-doc": [
        {
          "href": "https://docs.example.com/api/v2/",
          "type": "text/html"
        }
      ]
    },
    {
      "anchor": "https://api.example.com/payments/v1/",
      "item": [
        {
          "href": "https://api.example.com/payments/v1/openapi.yaml",
          "type": "application/vnd.oai.openapi+yaml;version=3.1"
        }
      ],
      "service-desc": [
        {
          "href": "https://api.example.com/payments/v1/openapi.yaml",
          "type": "application/vnd.oai.openapi+yaml;version=3.1"
        }
      ],
      "service-doc": [
        {
          "href": "https://docs.example.com/payments/",
          "type": "text/html"
        }
      ]
    }
  ]
}

In one request, the agent now knows:

• Every API the domain publishes
• Where to find each API's OpenAPI specification
• Where to find human-readable documentation
• The content type and version of each spec

Implementation

Serving an api-catalog is trivial — it's a static JSON file behind a well-known URI:

// Express.js
app.get('/.well-known/api-catalog', (req, res) => {
  res.type('application/linkset+json').json({
    linkset: [
      {
        anchor: 'https://api.yourapp.com/v1/',
        item: [{
          href: 'https://api.yourapp.com/v1/openapi.json',
          type: 'application/vnd.oai.openapi+json',
        }],
        'service-desc': [{
          href: 'https://api.yourapp.com/v1/openapi.json',
          type: 'application/vnd.oai.openapi+json',
        }],
        'service-doc': [{
          href: 'https://docs.yourapp.com/',
          type: 'text/html',
        }],
      },
    ],
  });
});

Or with nginx, just serve a static file:

location /.well-known/api-catalog {
    alias /var/www/api-catalog.json;
    default_type application/linkset+json;
    add_header Access-Control-Allow-Origin *;
}

Current adoption: Near zero. RFC 9727 is brand new. But its simplicity and the IETF backing make it the likely standard for API catalog discovery. Implementing it now is a 10-minute task that puts you ahead of 99% of the market.

---

Layer 3: OpenAPI specification (the workhorse)

If the api-catalog is the directory, OpenAPI is the detailed blueprint. An OpenAPI spec tells an agent everything it needs to make API calls: endpoints, parameters, request/response schemas, authentication methods, and error formats.

Why OpenAPI matters more for agents than humans

A human developer reads documentation, looks at examples, and figures out the API. They tolerate inconsistencies, missing details, and ambiguous descriptions.

An agent needs exact specifications. The difference between userId (camelCase) and user_id (snake_case) is the difference between a working API call and a 400 error. OpenAPI provides this precision.

What makes an OpenAPI spec agent-ready

Not all OpenAPI specs are created equal. Here's what agents actually need:

1. Detailed descriptions on every operation and parameter:

# ❌ Bad: agent has to guess what this does
paths:
  /users:
    get:
      operationId: getUsers

# ✅ Good: agent knows exactly what it gets
paths:
  /users:
    get:
      operationId: listUsers
      summary: List all users in the organization
      description: >
        Returns a paginated list of users. Results are sorted by 
        created_at descending. Use cursor-based pagination with 
        the 'after' parameter for large result sets.
      parameters:
        - name: limit
          in: query
          description: Maximum number of users to return (1-100, default 20)
          schema:
            type: integer
            minimum: 1
            maximum: 100
            default: 20
        - name: after
          in: query
          description: Cursor for pagination. Use the 'next_cursor' 
            value from the previous response.
          schema:
            type: string

2. Explicit authentication definitions:

components:
  securitySchemes:
    BearerAuth:
      type: http
      scheme: bearer
      bearerFormat: JWT
      description: >
        Obtain a token via POST /oauth/token using the 
        client_credentials grant type. Tokens expire after 1 hour.
    ApiKeyAuth:
      type: apiKey
      in: header
      name: X-API-Key
      description: >
        API key from your dashboard at https://app.example.com/settings/api.
        Prefix with 'sk_live_' for production, 'sk_test_' for sandbox.

security:
  - BearerAuth: []
  - ApiKeyAuth: []

3. Complete response schemas with examples:

responses:
  '200':
    description: Successfully retrieved users
    content:
      application/json:
        schema:
          type: object
          properties:
            data:
              type: array
              items:
                $ref: '#/components/schemas/User'
            pagination:
              type: object
              properties:
                next_cursor:
                  type: string
                  nullable: true
                has_more:
                  type: boolean
        example:
          data:
            - id: "usr_abc123"
              email: "agent@example.com"
              role: "admin"
              created_at: "2026-01-15T10:30:00Z"
          pagination:
            next_cursor: "eyJpZCI6InVzcl94eXoifQ=="
            has_more: true
  '429':
    description: Rate limit exceeded
    headers:
      Retry-After:
        schema:
          type: integer
        description: Seconds to wait before retrying
    content:
      application/json:
        schema:
          $ref: '#/components/schemas/RateLimitError'

4. Error response schemas (agents need to handle failures):

components:
  schemas:
    Error:
      type: object
      required: [error]
      properties:
        error:
          type: object
          required: [type, message]
          properties:
            type:
              type: string
              enum:
                - authentication_error
                - authorization_error
                - rate_limit_error
                - validation_error
                - not_found_error
                - server_error
            message:
              type: string
            param:
              type: string
              description: The parameter that caused the error
            docs_url:
              type: string
              format: uri
              description: Link to documentation about this error

Serving your OpenAPI spec

Make the spec available at a predictable, discoverable URL:

// Serve OpenAPI spec at a well-known path
app.get('/openapi.json', (req, res) => {
  res.type('application/vnd.oai.openapi+json')
     .sendFile(path.join(__dirname, 'openapi.json'));
});

// Also serve YAML version
app.get('/openapi.yaml', (req, res) => {
  res.type('application/vnd.oai.openapi+yaml')
     .sendFile(path.join(__dirname, 'openapi.yaml'));
});

// Link header on API responses pointing to spec
app.use('/api', (req, res, next) => {
  res.set('Link', '</openapi.json>; rel="service-desc"');
  next();
});

The Link header on API responses is a powerful signal. Even without prior knowledge of your API, an agent can discover the full specification by following the service-desc link relation from any API response.

---

Layer 4: llms.txt (the content map)

llms.txt is a simpler, more targeted standard than OpenAPI. Where OpenAPI describes API operations, llms.txt describes API content — documentation, guides, references — in a format optimized for language model consumption.

In our analysis of 10 popular API documentation sites, the best implementations follow a consistent pattern:

# Your Product

> One-line description that tells an LLM what this product does.

## Getting Started
- [Quickstart](https://docs.example.com/quickstart.md): 
  Set up your first integration in 5 minutes.
- [Authentication](https://docs.example.com/auth.md): 
  How to authenticate API requests with OAuth or API keys.

## API Reference
- [REST API](https://docs.example.com/api/rest.md): 
  Complete REST API reference with request/response examples.
- [Webhooks](https://docs.example.com/webhooks.md): 
  Configure event-driven integrations with webhook endpoints.
- [OpenAPI Spec](https://api.example.com/openapi.json): 
  Machine-readable API specification.

## SDKs
- [Node.js SDK](https://docs.example.com/sdk/node.md)
- [Python SDK](https://docs.example.com/sdk/python.md)

Key patterns from top implementations:

• Link to .md versions of pages when possible — agents can parse markdown directly without rendering HTML
• Annotate every link with a description — the annotation is often more useful to the agent than the linked page itself
• Organize by user journey — getting started → core concepts → API reference → SDKs
• Keep llms.txt as a summary and offer llms-full.txt for complete content

Adoption is growing fast. BuiltWith tracked over 844,000 websites with llms.txt as of October 2025, and the count has only increased since. Anthropic, Stripe, Cloudflare, Vercel, Supabase, Slack, and Linear are all early adopters.

---

Layer 5: Model Context Protocol (the agent layer)

MCP is the newest and most powerful layer. Introduced by Anthropic in late 2024 and adopted by OpenAI in March 2025, MCP defines a protocol for LLMs to dynamically discover, understand, and invoke tools — including APIs.

Where OpenAPI tells an agent "here's what endpoints exist," MCP tells an agent "here are the tools available, here's what each one does, and here's how to call them right now."

How MCP differs from OpenAPI

Aspect	OpenAPI	MCP
Purpose	Describe API structure	Enable LLM tool use
Discovery	Static spec file	Dynamic protocol
Invocation	Agent builds HTTP requests	MCP server handles execution
Authentication	Described, not handled	Managed by the server
Context	Endpoint-level	Task-level (what the tool does)

Converting OpenAPI to MCP

The most practical path for existing APIs is to generate MCP tool definitions from your OpenAPI spec. A research paper from July 2025 (arXiv:2507.16044) demonstrated this approach, and several tools now automate the conversion.

Here's what a simple MCP server looks like that wraps an existing REST API:

import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
import { StdioServerTransport } from '@modelcontextprotocol/sdk/server/stdio.js';
import { z } from 'zod';

const server = new McpServer({
  name: 'your-api',
  version: '1.0.0',
});

// Define a tool that maps to an API endpoint
server.tool(
  'list_users',
  'List all users in the organization. Returns paginated results.',
  {
    limit: z.number().min(1).max(100).default(20)
      .describe('Maximum number of users to return'),
    after: z.string().optional()
      .describe('Pagination cursor from previous response'),
  },
  async ({ limit, after }) => {
    const params = new URLSearchParams({ limit: String(limit) });
    if (after) params.set('after', after);

    const response = await fetch(
      `https://api.yourapp.com/v1/users?${params}`,
      {
        headers: {
          'Authorization': `Bearer ${process.env.API_TOKEN}`,
          'Content-Type': 'application/json',
        },
      }
    );

    if (!response.ok) {
      return {
        content: [{
          type: 'text',
          text: `Error: ${response.status} ${await response.text()}`,
        }],
        isError: true,
      };
    }

    const data = await response.json();
    return {
      content: [{
        type: 'text',
        text: JSON.stringify(data, null, 2),
      }],
    };
  }
);

// Start server
const transport = new StdioServerTransport();
await server.connect(transport);

The power of MCP is that the LLM doesn't need to understand HTTP, headers, authentication, or URL construction. It sees a tool called list_users with typed parameters and a description. The MCP server handles everything else.

MCP + OpenAPI: the practical architecture

For most SaaS companies, the pragmatic approach is:

1. Publish an OpenAPI spec (for existing API clients and developer tools)
2. Run an MCP server that wraps your API (for LLM agents)
3. Use tools like Gravitee or Xano that auto-generate MCP servers from OpenAPI specs

Agent (LLM)
    ↓
MCP Protocol
    ↓
MCP Server (your wrapper)
    ↓
REST API (your existing endpoints)

The OpenAPI Initiative's February 2026 newsletter announced a Special Interest Group focused specifically on making OpenAPI specs "agent-ready" for LLM use, including capability discovery and intent signaling. The standards are converging.

---

The full stack: putting it all together

Here's a complete implementation of all five layers for a SaaS API:

1. robots.txt

User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xml

2. /.well-known/api-catalog

{
  "linkset": [{
    "anchor": "https://api.example.com/v1/",
    "service-desc": [{
      "href": "https://api.example.com/v1/openapi.json",
      "type": "application/vnd.oai.openapi+json"
    }],
    "service-doc": [{
      "href": "https://docs.example.com/",
      "type": "text/html"
    }]
  }]
}

3. /openapi.json

Your complete API specification (see examples above).

4. /llms.txt

# Example API

> Example provides a REST API for managing users, teams, and 
> projects. Full OpenAPI spec at /openapi.json.

## Authentication
- [Auth Guide](https://docs.example.com/auth.md): 
  OAuth 2.0 Client Credentials for agent access.

## API Reference
- [Users](https://docs.example.com/api/users.md): CRUD operations
- [Teams](https://docs.example.com/api/teams.md): Team management
- [OpenAPI Spec](https://api.example.com/v1/openapi.json): 
  Machine-readable specification

5. MCP Server

Published as an npm package or hosted service that agents can connect to.

Discovery flow for an agent

With all five layers in place, an agent's discovery process looks like:

1. GET /robots.txt          → "Am I allowed?"         (200, yes)
2. GET /llms.txt            → "What does this do?"    (200, summary)
3. GET /.well-known/api-catalog → "Where are the APIs?" (200, catalog)
4. GET /openapi.json        → "How do I call them?"   (200, full spec)
5. Connect via MCP          → "Let me use the tools"  (connected)

Five requests. Zero guessing. Total time: under 2 seconds.

Compare that to the status quo: 15+ requests, HTML parsing, JavaScript rendering, trial-and-error API calls, and a 50/50 chance of giving up.

---

Implementation priority: what to do first

Not every SaaS company needs all five layers on day one. Here's the priority order based on impact per effort:

Start here (30 minutes)

1. Publish llms.txt — highest adoption momentum, lowest effort
2. Ensure robots.txt doesn't block AI crawlers or your docs paths
3. Add a Link header to API responses pointing to your spec

Next step (half day)

4. Publish your OpenAPI spec at a discoverable URL (/openapi.json)
5. Implement /.well-known/api-catalog per RFC 9727

When you're serious (1-2 weeks)

6. Build or generate an MCP server that wraps your API
7. Add agent-optimized descriptions to every OpenAPI operation
8. Publish .md versions of documentation pages

Measuring progress

Each of these steps improves your agent-readiness score. In our data, companies with discoverable API specifications score an average of 14 points higher than those without. Companies with llms.txt score 14 points higher. The effects compound — implementing the full stack can swing a score by 25-35 points.

---

The future: agents discovering agents

The discoverability standards described here are designed for a world where agents find and use human-built APIs. But the next wave is agents discovering each other — automated negotiation of capabilities, pricing, and access between autonomous systems.

The Model Context Protocol already hints at this with its tool discovery mechanism. An MCP client (an agent) connects to an MCP server (another agent or API wrapper) and dynamically discovers what tools are available. Extend this to a network where agents publish their capabilities to a shared registry, and you get a machine-to-machine marketplace.

RFC 9727's api-catalog is the first step. MCP's tool discovery is the second. The third — a decentralized registry where agents advertise capabilities and discover each other — is being built right now by companies like AgentGate.

The companies that make their APIs discoverable today will be the ones agents recommend tomorrow. The standard stack is here. The implementation is straightforward. The only question is whether you build for the agents now or scramble to catch up later.

---

How discoverable is your API? Run your domain through the AgentGate benchmark to see your agent-readiness score, including discoverability signals.