robots.txt

How a crawler reads it

The rules are in RFC 9309. A crawler fetches /robots.txt, looks for the group whose User-agent line matches its own product token, falls back to the * group if there is none, and treats everything as allowed if neither exists. Inside the group the rule that matches the most characters of the path wins, and when an allow and a disallow match equally, allow wins. A 4xx response means the crawler may fetch anything; a 5xx means it must assume the whole site is disallowed. Crawlers have to parse at least 500 kibibytes, so a very long file is not a defence either.

What it cannot do

Two things owners expect of it that it does not do. It does not keep a page out of an index: Google’s documentation says a URL disallowed in robots.txt can still be indexed from links elsewhere, and that noindex or password protection are the tools for that. And it is not access control. The RFC says so outright: a path listed there is a path announced to anyone who reads the file.

AI crawlers and their tokens

Each AI company runs more than one agent, each with its own token, so a rule written for one says nothing to the others. OpenAI’s crawler documentation lists OAI-SearchBot, which surfaces sites in ChatGPT search; GPTBot, which collects training data; and ChatGPT-User, which fetches when a user asks. Disallowing OAI-SearchBot removes a site from ChatGPT search answers, while disallowing GPTBot only signals that content should not be trained on. Perplexity’s documentation says PerplexityBot respects robots.txt and that Perplexity-User, the fetcher for a user’s direct request, generally does not.

In this site’s own audit of 9,483 small-business sites, 3.2% disallowed AI crawlers in robots.txt. A further 7.2% sat behind bot protection that served a challenge page instead of the site, which shuts a crawler out without a line in any file. The figures are in what 9,483 small business websites serve to an AI crawler.