llms.txt

llms.txt is a proposed convention: a plain-text file served at /llms.txt that summarizes a website and lists the pages a language model should read, in a form easier for a model to take in than the site’s HTML. It is a proposal, not a standard any assistant has committed to honoring.

What the file is for

The idea is a reading list. The file names the site, says in a sentence or two what it is, and points at the pages that matter with a line of description each, so a model with a limited context window is directed to the right text instead of parsing menus, footers and scripts. The format was set by its proposers rather than a standards body, and no assistant vendor documents reading it. This site serves one at /llms.txt; it points at the research article and the methodology.

llms.txt and robots.txt

They sit next to each other at the root and do opposite things. robots.txt is defined by RFC 9309, is read per product token under a longest-match rule, and tells a crawler what it may not fetch; the crawlers behind the assistants document that they read it. llms.txt is advisory, tells a model what it should read, and has no documented reader. A site that blocks an AI crawler in the first file has made the second irrelevant to that crawler.

How common it is

In this site’s own audit of 9,483 small-business sites, 63.6% served no /llms.txt at all. The finding is reported last in what 9,483 small business websites serve to an AI crawler because it is the one whose weight is unknown: it costs an hour, it is a clean sign that someone has thought about machine readers, and there is no evidence yet that an assistant acts on it.

A common question

Is it worth serving an llms.txt file if no assistant documents reading it?

The entry puts it as an open question rather than a settled one. Nobody has published evidence that an assistant acts on the file, and no vendor documents reading it, so its weight is unknown. Against that, it takes about an hour and it is a clear sign that someone has considered machine readers. This site's audit of 9,483 small-business sites found 63.6% serving none at all, and the finding is reported last for exactly that reason.

Sources

robots.txt semantics: RFC 9309. The 63.6% figure is this site’s own, computed from its audit records; sample drawn per metro and trade, not from a random frame.