Opening /llms.txt on your own domain takes ten seconds. On most sites it returns a 404. The question that follows is always the same: if we add this file, will ChatGPT start citing us? The honest answer is no. But showing where the question goes wrong is more useful than describing the file — because llms.txt is not a ranking lever, it is a readability decision.
What is llms.txt?
llms.txt is a proposed file that sits at the root of a site and hands language models a plain-text map of its content. The address is fixed: yourdomain.com/llms.txt. The format is markdown — a single H1, a one-sentence definition, then lists of links under section headings. Jeremy Howard put the proposal forward in September 2024; the technical definition lives at llmstxt.org and no standards body has ratified it to date.
It is easier to say what it is not. robots.txt is a permission file: it states which paths must not be crawled. sitemap.xml is an inventory: it lists every URL and says nothing about what any of them contain. llms.txt does a third job — it names the pages that matter and, in one line each, what question they answer. That is the difference: a table of contents written for a machine.
Why did this file appear?
Two technical constraints. First: most AI crawlers do not execute JavaScript, so a page whose content is assembled in the browser arrives at them as an empty shell. Second: a language model's context window is finite, and the budget it can spend reading your site drains into menus, cookie notices, footer links and repeated boilerplate.
llms.txt clears both in one move. Being plain text, it needs no JavaScript; being short, it barely dents the context window. HTML is written for a browser and markdown is written for reading — and reading is the only thing a model actually does on your page.
The constraint itself is not new. What is new is that the party running into it is a program rather than a person. That is why the AI search optimisation conversation opened a third front after speed and indexability: what can the machine actually extract from your page?
The format: a markdown list of links
The format is deliberately narrow. llmstxt.org describes a single layout, and five elements carry the whole file:
- One H1: the name of the site or the organisation. No second H1 appears in the document.
- Directly beneath it, a one-sentence definition as a blockquote: what the organisation does and who it works for.
- Sections marked by H2 headings: services, products, documentation, case studies, contact. Section names are free-form.
- Link lines under each section — page name in square brackets, full URL in parentheses, a one-sentence description after a colon. The description is optional, and it is where the file's value actually sits.
- An optional Optional section: secondary links the model can skip when context runs short.
URLs must be absolute. A file with relative paths loses its meaning the moment it leaves its own site — and the place a model reads your file is almost never your site.
What is the difference between llms.txt and llms-full.txt?
The division of labour is clear: llms.txt is a map, llms-full.txt is an export. The map routes the model to the right URL and weighs a few kilobytes. The export carries the body text of those pages in one file and can run into hundreds of kilobytes; its purpose is to let a model describe you accurately without crawling the site at all.
llms-full.txt was not part of the original proposal's core; it grew out of documentation tooling and spread from there. Bigger is not automatically better: a file that exceeds the context window gets truncated, and you do not get to choose which half survives. The practical rule is to publish both and to keep the export limited to content you genuinely want quoted.
Do search engines actually read this file?
As things stand, no major search engine or model provider has announced official support for llms.txt. Statements from Google's side indicate the file plays no part in ranking, and neither OpenAI, Anthropic nor Perplexity has published a declaration of support. If an agency tells you otherwise, ask them for the source.
On the publishing side, though, adoption is moving fast. Companies that ship developer documentation have made the file a de facto standard, and most documentation platforms now generate llms.txt and llms-full.txt automatically. The file therefore sits in an asymmetry: plenty of sites write it, none of the engines admit to reading it. There is also no reason to assume the asymmetry is permanent — on the day an engine announces support, whoever already has the file does nothing at all.
Where the file does land today is elsewhere: agents that fetch live pages on a user's request. Crawlers like ChatGPT-User, Claude-User and Perplexity-User pull the URL at the moment they answer, and a clean map at the root is processed faster than thirty pages of HTML. The same file serves coding and documentation assistants, the chat assistant you build yourself, and any retrieval setup feeding on your content.
Which makes the decision frame simple: it costs an hour, the risk is close to zero — the file lists pages that are already public and exposes nothing new — and the return is uncertain but positive. This is not work for a quarterly roadmap. It is work for an afternoon.
How do you write an llms txt file, step by step?
It is a six-step job. Writing the file by hand is possible, but getting the first decision right makes the rest easier: who is this file for, and which question should it answer?
- Define the reader. What should a model name you for — your services, your technical documentation, your case studies? The file's sections follow from that answer.
- Choose the pages. Not the whole site — list the twenty to forty pages a model needs in order to describe you correctly. The sitemap already provides the inventory; here you are making a selection.
- Write a one-sentence description on every line. Do not repeat the page title; state which question the page answers.
- Write the definition sentence. The single line under the H1 is the sentence a model will reach for when it describes you. Write the sentence you would want to read about yourself.
- Generate the file from your content source. A hand-written list goes stale at the first slug change, and nobody notices.
- Publish it and verify it. The content type should be text/markdown or text/plain; a file that triggers a download in the browser is being served with the wrong header. If your site is multilingual, give every language its own version.
There is no seventh step. Once the file is live, the remaining work is making the content itself quotable, and llms.txt does not stand in for that.
How did we build llms.txt on our own site?
Four files are published on this site: /llms.txt, /tr/llms.txt, /en/llms.txt and /llms-full.txt. None of them contains a hand-written link line. All four are assembled from the content layer — the service, case, package, article and consultant records — turned into static files at build time and served as text/markdown. None of them is computed per request.
The reason is maintenance. When a slug changes, a hand-written llms.txt goes stale silently: the file still returns 200, the links inside it return 404, and your reader is not the first to notice. Because the link lines are produced from the content objects, that scenario cannot occur — a service URL comes from the service's own slug field, a case URL from the case's. The domain is not hard-coded either; it is read from a single site-address constant.
What the file holds today: 12 services across three disciplines, 9 case studies, every published article, contact details and five resource pages — over forty links in the Turkish version, each with its own one-sentence description. The root /llms.txt merges both languages into one document, Turkish first, English after a separator. The single-language versions carry no links to the other language, because a single-language file's job is to map its own language.
/llms-full.txt goes further: the scope and out-of-scope lines of every service, the duration and fixed price in three currencies for four packages, and the problem-approach-outcome narrative of nine case studies with their metrics attached to context. On the SIM Printing Suppliers line, the 15× lift in organic traffic sits next to the note "in 6 months; via replatforming and the content programme". We never print a number without its frame — if a model is going to quote it, it should quote the frame too.
We do not claim these files produced the results in the SIM Printing Suppliers case. There, organic traffic growing 15× in six months and visibility in AI engines going from zero to 40,000 came from rebuilding the site in five languages and writing the content on a question-and-answer architecture. llms.txt is a footnote to that work, not its lever. The order is deliberate: content first, map second.
What should robots.txt say to AI crawlers?
For llms.txt to be read at all, the site has to be open to that crawler, and robots.txt is where the decision gets made. Our robots file names ten AI crawlers explicitly and allows every one of them: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended and CCBot.
Access was already open under the wildcard block; naming each crawler is about making the intent legible. There is a trap in the mechanics, though: in robots.txt the most specific user-agent block wins, so a named bot never reads the wildcard block at all. If you do not repeat the disallow list covering app, admin and API paths inside every block, the crawlers you named end up with broader access than everyone else. We repeat the same disallow list across all eleven blocks.
One more detail: non-production environments return Disallow: / to every crawler. An indexed preview domain competes with the canonical version, and that is one of the faults that quietly undoes an otherwise well-built GEO programme.
What are the most common mistakes?
Publishing the file is easy, and so is publishing it wrong. Five mistakes recur in the field.
- Publishing a copy of the sitemap. A list of URLs without descriptions already exists in sitemap.xml; the descriptions are llms.txt's only contribution.
- Writing it by hand and forgetting it. Six months on, the file lists pages the site no longer has, and the site has pages the file never heard of.
- Serving it with the wrong content type. An llms.txt returned as text/html is not processed by anything expecting markdown.
- Publishing one language on a multilingual site. Your English pages simply never appear on the model's map.
- Mistaking the file for a ranking promise. llms.txt persuades nobody to cite you; the content does that — and appearing in Google AI Overviews is a line of work in its own right.
Which test can you run this week?
The thesis: llms.txt wins you no ranking, but it makes your readability debt visible. The question you ask while assembling it — which forty pages would let a model describe me correctly — goes unasked for years on most sites, and the real gain is in the answer.
The concrete test you can run today takes two minutes. Type your own domain into the address bar and add /llms.txt. If you get a 404, ask the second question before you build the file: which pages would you put in it, and what would you write beside each one? If that list is hard to produce, the problem is not the file — it is how the site explains itself. Then run the same test on a competitor's domain; ten seconds shows you who has done the work.
Once the list exists, the next job is the content itself. Our guide to standing out in AI search walks through that part step by step, and the numbers from where it has been built sit on our case pages.