llms.txt explained: what it is and whether it matters
A proposed standard for pointing language models at your best content. Adoption is not universal and the honest answer about its value is nuanced. Here is what it does and does not do.
Key takeaways
- llms.txt is a proposed convention, not a standard, and it grants no access and blocks nothing.
- It is neither a sitemap nor a ranking factor; Google has not indicated it reads the file.
- The full-text companion file is the more useful half, giving retrieval clean prose without markup noise.
- Add it because it is cheap, but prioritise crawler access, rendering and sourced answers ahead of it.
llms.txt is a proposed convention: a markdown file at the root of your domain that lists your most useful pages with short descriptions, so a language model retrieving your site has a curated map rather than a crawl. It is not an official standard, adoption by major model providers is not universal, and it does not control access. It is a hint, not a directive.
What it is not
Three confusions are worth clearing up. It is not robots.txt; it grants and denies nothing, and a crawler ignoring it is not violating anything. It is not a sitemap; a sitemap enumerates every URL for a crawler, while llms.txt curates a small set for a reader. And it is not a ranking factor in Google Search, which has not indicated that it reads the file at all.
Anyone telling you that adding llms.txt will get you cited is overstating a file whose principal benefit is cheap and indirect.
Why it is still worth adding
- It costs almost nothing. A single generated file, updated alongside your sitemap, with no ongoing maintenance burden.
- It forces a useful decision. Choosing the fifteen pages that best represent your business is an exercise most teams have never done, and the answer often reveals that the most important pages are not the ones getting attention.
- The companion file is the real prize. A full-text version, commonly llms-full.txt, gives a retrieval system clean prose without navigation, cookie banners or markup noise, which is materially easier to quote accurately.
- If adoption grows, you already have it. The downside of being early here is a few kilobytes.
What to put in it
Keep it short and honest. A title, one sentence describing what the organisation does, then a small number of linked sections: your core service pages, your best explanatory content, and your contact or pricing information. Write the descriptions for a reader who knows nothing about you. Do not list every blog post; a curated file that lists two hundred URLs has become a bad sitemap.
Generate it from the same source that generates your sitemap so it cannot drift out of date. A file describing pages that no longer exist is worse than no file, because it wastes a retrieval attempt and signals neglect.
Where to spend the effort instead
If you have a limited amount of time for generative optimisation, llms.txt should be the last thing on the list, not the first. Ahead of it: making sure AI crawlers are not blocked, rendering content server-side, putting direct answers at the top of pages, adding sourced figures with dates, and making your entity data consistent. Those have demonstrated effects. llms.txt has a plausible future one.
Add it because it takes an hour, not because it will change your visibility this quarter.
Sources