How llms.txt works
llms.txt is a proposed plain markdown file placed at the root of a domain, in the same position as robots.txt. Instead of telling crawlers what they may not touch, it is meant to do the opposite: point a language model at the pages that matter, with a line of context for each, so a model reading your site gets a curated map rather than a raw crawl.
The format is deliberately simple — a heading, an optional summary, then linked lists of the documents you consider canonical. Some sites publish a longer companion file holding the full text of those pages in one place. Nothing about it is enforced. It is a file you choose to write and a machine chooses whether to read.
Why llms.txt matters
The problem behind it is genuine. A model retrieving from a website meets navigation, cookie notices, related-post blocks and template furniture before it reaches the sentence that answers the question, and a short list of canonical documents would cut that noise considerably.
What matters just as much is what has not happened. No major search engine or assistant has confirmed that it reads the file, and Google’s search representatives have said publicly that Google does not use it. Any claim that llms.txt improves visibility in AI search should be treated as unverified. It is a proposal with a following, not a supported standard.
Where llms.txt goes wrong
The expensive failure is duplication. Publish one by hand and you now maintain a second, summarised version of your site, which begins drifting the day a price, a service or a URL changes. If anything ever does read it, it reads your out-of-date copy with exactly the confidence it would give your live pages.
The other failure is displacement. Hours spent writing an llms.txt are hours not spent on the things demonstrably read: the page text itself, structured data, internal links, and a site that states its facts without needing JavaScript. It also restricts nothing — access control belongs in robots.txt and on your server, never here.
What to do about it
My advice is short: publish it only if you can generate it. If your site can produce the file automatically from the same content that builds your pages, the upkeep is close to nothing and the experiment is harmless. If it means somebody editing markdown by hand every month, skip it and spend the time on the pages.
Either way, follow the evidence rather than the enthusiasm. Watch your server logs for requests to the file, so you know whether anything actually fetches it, and judge your standing in AI answers by putting your customers’ questions to the assistants themselves. If a supplier offers llms.txt as a deliverable with a promised result, ask what they will show you to prove it worked.