Jeremy Howard at Answer.AI proposed llms.txt in late 2024 as a markdown-formatted index at the root of a domain, designed to give large language models a curated entry point into your site. The idea is straightforward. Robots.txt tells crawlers where they may not go. Sitemaps tell crawlers what exists. llms.txt tells an LLM what matters and in what order, written in prose a model can parse cheaply. It is a proposal, not a web standard, and that distinction matters.
Adoption as of 2026 is real but uneven. A growing number of documentation sites, SaaS marketing sites, and developer-tool brands publish llms.txt files. Anthropic, Cloudflare, Vercel, and Stripe have versions. But there is no public confirmation from OpenAI, Google, or Anthropic that their crawlers read the file as a primary signal. ChatGPT-User, GPTBot, ClaudeBot, PerplexityBot, and Google-Extended still mostly behave like traditional crawlers backed by sitemaps and link graphs. Treat llms.txt as a complement, not a replacement.
The file itself is simple. A top-level H1 with your site name, an optional blockquote summary, then sections of markdown links grouped by purpose. Documentation here, blog posts there, API references in a third group. Each link points to a clean URL, ideally one that returns either markdown or a print-friendly HTML version. The companion file, llms-full.txt, is the same idea but dumps the actual content inline so a model can ingest the whole corpus in one fetch. That is useful for retrieval-augmented systems and zero-shot extraction.
The common misunderstanding is that publishing an llms.txt file will make ChatGPT or Claude cite you more. It will not, at least not directly. What it does is make you legible. If a model or an AI agent does decide to crawl your site, the file removes ambiguity about which URLs are canonical, which are marketing fluff, and which contain the actual reference material. For an agent doing structured retrieval, that saves tokens and reduces hallucination. For a generic chatbot, it is mostly invisible today.
So why bother. Two reasons. First, the cost is genuinely low. If your site already has clean URLs and decent information architecture, generating llms.txt is a five-minute job and a one-line change to your sitemap. Second, the trajectory is one-way. AI search is taking share from traditional search every quarter. Sites that are easy for agents to parse will, on balance, be cited more often as that traffic compounds. The bet is asymmetric. Small effort, plausible upside, no downside.
Create a curated llms.txt for AI crawlers.
- What is llms.txt?
- A proposed standard that tells AI crawlers what pages of your site are useful for training and inference.
- Where do I host the file?
- At the root of your domain — /llms.txt — alongside robots.txt.
- Will major LLMs respect this file?
- Adoption is early but growing. ChatGPT, Claude, and several open-source models reference llms.txt during retrieval.
- How is llms.txt different from robots.txt and sitemap.xml?
- Robots.txt is a directive that restricts crawler access. Sitemap.xml is an exhaustive machine-readable list of URLs for indexing. llms.txt is a curated, human-readable markdown document for AI models, organised by editorial priority rather than completeness. The three serve different audiences and should coexist, not replace each other.
- What is the difference between llms.txt and llms-full.txt?
- llms.txt is the index. A short markdown file with links grouped by section. llms-full.txt is the same structure but with the actual page content inlined as markdown, so a model can ingest the whole corpus from a single fetch. Use llms.txt for navigation and llms-full.txt when you want zero-friction retrieval of your reference material.
- Does publishing llms.txt change my Google rankings?
- No. Google's organic ranking algorithm does not use llms.txt as a signal as of 2026. It might influence how Google's AI Overviews surface content if Google-Extended chooses to read the file, but there is no confirmation of that behaviour. Do not publish llms.txt expecting a traditional SEO lift. The upside is in AI-driven discovery, not blue-link rankings.
- Can llms.txt block AI crawlers from training on my content?
- No. That is robots.txt territory, plus the noai and noimageai meta tags, plus the AI-specific user agents you list. llms.txt has no directive language and no enforcement. It is an invitation to read curated content, not a restriction on training data. If you want to block training, use the proper tools and assume partial compliance.
- Should I publish llms.txt if my site is small or new?
- Probably yes, if the cost is genuinely low for you. A site with twenty good pages benefits more from a clean five-link llms.txt than a thousand-page site dumps in a sprawling file. The format rewards small, opinionated, well-curated sites. If your information architecture is already a mess, fix that first. llms.txt does not paper over bad IA.
- How often should I update llms.txt?
- Quarterly is a reasonable cadence for most sites. Monthly if you publish heavily or your reference content turns over fast. Tie the update to the same review cycle as your sitemap and internal links audit. The file should reflect what you currently want models to know about you, not what mattered eighteen months ago.
- Does llms.txt need to be at the root of the domain?
- Yes. The proposal specifies /llms.txt at the domain root, the same convention as robots.txt and security.txt. Subdirectories or subdomains can technically host their own files, but discovery tooling and the convention itself assume the root path. Put it at example.com/llms.txt and link to it from your sitemap or robots.txt for good measure.
Publishing llms.txt today is a small bet on a future that is arriving unevenly. The standard might harden, get superseded, or quietly fade. None of that changes the fact that a curated, agent-readable index of your best content is good hygiene regardless of which crawler ends up reading it. Ship the file, keep it honest, and move on.
