llms.txt for API docs, and what AI crawlers actually read
A file at your docs root that tells a model what your product is and where to look. Cheap to publish, easy to get wrong, and not a substitute for the thing it points at.
llms.txt is a markdown file at the root of your site that describes what the site is and links to the pages worth reading. The convention is young, the cost of publishing one is close to zero, and for API documentation it is unusually well suited, because API docs are exactly the kind of content a model is asked about and exactly the kind of content that is painful to crawl.
It is also widely misunderstood in two directions at once. Some teams treat it as a magic ranking file. Others dismiss it because no crawler is contractually obliged to read it. Both miss what it is actually for.
What it is
A map, in markdown, at /llms.txt. An H1 with the name, a blockquote summarising what the thing is, then link lists under headings, each link followed by a short description of what a reader will find there.
# Acme
> Payments API for marketplaces. REST, OpenAPI 3.1, SDKs in six languages.
## Products
- [Payments API](https://acme.dev/payments): Charges, refunds, disputes.
## Documentation
- [Quickstart](https://docs.acme.dev/quickstart): First call in under ten minutes.
- [Authentication](https://docs.acme.dev/auth): Key types, scopes, rotation.That is the whole format. Its value is that it answers "what is this and where should I look" in a few hundred tokens, rather than requiring a model to infer it from a rendered marketing page full of navigation.
Why it fits API docs specifically
Three properties of API documentation make this worth more than it is for a typical site.
Your content is large and repetitive. A reference has hundreds of pages that differ in the details and look identical in structure. A crawler landing in the middle of one has almost no signal about what the product is.
Your content is the answer to questions people actually ask assistants. "How do I authenticate with X", "does X support webhooks", "what is the rate limit on X" are the questions, and your docs are the answer. Being legible to the thing being asked is not a marginal gain.
Your content changes. An assistant working from a training-set memory of your API is working from a version you have moved on from. A file that points at your current docs is a correction mechanism.
What to put in it, for an API
Lead with what the API is, in one sentence a model can quote. Not a tagline. "Payments API for marketplaces" beats "the future of money movement".
Then the pages that answer the common questions, in this order: getting started, authentication, the reference root, errors, rate limits, webhooks, SDKs, changelog. That order is roughly the order a new integrator needs them, and it is a reasonable proxy for what an assistant will be asked.
Describe each link in terms of the question it answers. "Authentication: key types, scopes and rotation" is useful. "Authentication" twice is not.
Include your changelog. It is the page that tells a model your API has moved since whatever it remembers, and it is the one most people forget to list.
Keep it short. The value is that it is small enough to read entirely. A file listing four hundred endpoints has become the problem it was meant to solve.
llms.txt and llms-full.txt are different jobs
The convention has a second file, and conflating them is the common mistake.
llms.txt is the index: links and one-liners, a few hundred tokens, meant to be read whole and then followed.
llms-full.txt is the content: the actual documentation inlined, which for a real API can be enormous. It exists for the case where a tool wants everything in one fetch, and it is genuinely useful for a small product and genuinely unusable for a large one.
Publish both if you can, and keep them honest about their roles. What you should not do is publish an llms.txt that is secretly full text, because then the cheap file is expensive and nothing is a map.
What it does not do
It does not make a model use your documentation. Nothing does. Adoption is inconsistent, no major crawler has committed to it as a standard, and treating it as a ranking mechanism is the same category error as keyword-stuffing a meta tag.
It also does not fix thin documentation. A perfect index pointing at endpoints with no descriptions gets an assistant to a page that cannot answer the question. The file is a pointer; the pointer is worth what it points at.
And it is not the strongest tool available for API docs specifically. If you want an assistant to work from your current documentation reliably, giving it a way to query your docs directly beats hoping it fetched a file. An MCP server does that: the agent searches your documentation, reads one endpoint in full, checks your changelog, and gets the real method names from your SDK, instead of reading a summary and guessing the rest.
The honest framing is that llms.txt is passive and broad, MCP is active and precise, and they are not competing. Publish the file because it costs an afternoon. Wire up the server because it is what changes the output.
Generate it rather than maintain it
The failure mode of any hand-written index is drift. A file that lists pages which have since moved is worse than no file, because it is confidently wrong.
If your docs are generated from a spec, the index should be too. Ours is: the marketing site and every customer docs site emit llms.txt from the same source the navigation comes from, so a page that moves moves in both. That is the whole reason it stays accurate, and it is the property to look for whether you build it yourself or not.
If you want to see the shape, octri.dev/llms.txt is ours, and every docs site we host serves one at its own root.
