robots.txt for AI crawlers: what to allow, and what each token controls

How does an AI crawler read robots.txt?

The file is read the way RFC 9309 says, and every AI crawler whose operator publishes documentation says it follows that standard. Rules come in groups, and each group opens with one or more User-agent lines naming a product token. A crawler looks for the group whose token matches its own most specifically. If it finds one, it obeys that group and only that group. If it finds none, it falls back to the group for *. If there is no file at all, or the file is empty, everything is allowed.

Inside a group the most specific matching path wins. Where an Allow and a Disallow match a URL equally, the allow wins. So Disallow: /private/ followed by Allow: /private/prices keeps the folder closed and the price page open, whichever order the two lines appear in.

The consequence people miss is the fallback rule. A crawler that finds a group addressed to its own token ignores the * group entirely; the two are not merged. If your * group disallows /admin/ and you then add a group for OAI-SearchBot that says only Allow: /, OAI-SearchBot may now fetch /admin/. Anything you mean to keep from everyone has to be repeated inside every group you write.

Are AI crawlers blocked by default?

No. A site with no robots.txt, or one whose file names no AI token and disallows nothing under *, admits every crawler that reads the file. Most small business sites are in that state. In our audit of 9,483 small business websites (40 trades, 90 metros, September 2026), 3.2% disallowed AI crawlers in robots.txt on purpose.

What does block AI crawlers without anyone writing a rule is bot protection. 7.2% of the sites in the same audit served our crawler a challenge page instead of the site. The challenge is issued by the CDN or a security plugin, it never consults robots.txt, and it stops an assistant's fetch as completely as a Disallow line would. The difference is that nobody decided it.

Two more things sit outside the file. The fetchers that act on a person's request are not crawlers in the standard's sense: Perplexity documents that Perplexity-User generally ignores robots.txt because a user asked for the page. And robots.txt has no vocabulary for what happens to a page after it is fetched. Whether the text goes into a training set or a search index is decided by which token fetched it, which is the subject of the next two sections.

The nineteen tokens and the job each one does

Each AI company runs more than one crawler, and the token in the user agent says which job a fetch belongs to. The split that matters for being named in an answer is between crawlers that feed answers and crawlers that feed training. This is the table our crawler access check reads a robots.txt against; the descriptions are the operators' own, shortened.

Three consequences are stated plainly in the operators' documentation. OpenAI: disallowing OAI-SearchBot removes a site from ChatGPT search answers, and disallowing GPTBot does not. Google: Googlebot feeds Search, and the separate Google-Extended token controls whether crawled content is used for Gemini training and grounding, with no effect on whether a page appears in Search. Perplexity: PerplexityBot indexes, and Perplexity-User fetches for one person and generally does not consult the file.

Does blocking GPTBot keep a site out of ChatGPT?

No. GPTBot collects text that may be used to train OpenAI's models. It has nothing to do with ChatGPT search, which is built by OAI-SearchBot, or with what ChatGPT can open when a user pastes your URL, which is ChatGPT-User's job. A robots.txt that disallows GPTBot and nothing else leaves both of those exactly where they were. If the intention is to leave ChatGPT's search results, the token to disallow is OAI-SearchBot.

The same split exists at every operator that publishes one. Google keeps Googlebot apart from Google-Extended. Perplexity keeps PerplexityBot apart from Perplexity-User. Anthropic's help center names three: ClaudeBot on the training side, and Claude-SearchBot and Claude-User on the answer side, and it says all three honor robots.txt, which OpenAI does not promise for ChatGPT-User and Perplexity does not promise for Perplexity-User. A site that wants to be cited needs the answer-side tokens to reach its pages. What it decides about the training-side tokens is a separate question.

We take no position on that second question beyond one: make the decision on purpose. A shape we saw often in the audit's robots files: rules written years ago for Googlebot alone, plus one GPTBot line added in 2023 because a blog post said it would keep the business out of ChatGPT. It did not, and it did not need to. Decide which side you are ruling on, then write the rule for the tokens on that side.

A robots.txt that lets assistants in and leaves training to you

This file admits every answer-side crawler, opts out of every training-side crawler, and keeps the usual private paths closed for everyone. Swap the training group's Disallow: / for an empty Disallow: if you would rather allow training too. Either way the answer-side group is what keeps you citable.

# Crawlers that feed answers: allowed everywhere except the private paths
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Googlebot
User-agent: Bingbot
User-agent: Applebot
User-agent: DuckAssistBot
Allow: /
Disallow: /admin/
Disallow: /account/

# Crawlers that collect training data: this example opts out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
User-agent: Bytespider
Disallow: /

# Everyone else
User-agent: *
Disallow: /admin/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Three notes on the shape. A group may open with several User-agent lines, and they all share the rules under them, so the answer-side crawlers are one group rather than ten. The private paths appear twice because of the fallback rule from the first section: a token with its own group never reads the * group. And the file is plain text at /robots.txt on the exact host the site serves from; a copy on www does nothing for the bare domain, and the other way round.

Google's own guidance is worth keeping in view while you edit. Its page on AI features says no special markup or file is required for AI Overviews or AI Mode: a page has to be indexed and eligible for a snippet, which is what Googlebot's access buys you. robots.txt is a permission, not a signal. Opening the door does not make an assistant walk through it; only what it finds on the other side does that.

What robots.txt cannot fix

It governs crawling, not indexing. Google's robots.txt introduction says a URL blocked in the file can still be indexed if other pages link to it, and its page on noindex adds the trap that follows: a noindex tag on a page the crawler is not allowed to fetch is never read. Keeping a page out of results means letting the crawler in to see the tag.

It does nothing about pages that are empty until a script runs. Our crawler executed no JavaScript, which is how most fetchers behave, and 3.0% of the audited sites gave it an empty shell where the site should have been. Google renders JavaScript, but its documentation describes rendering as a separate, deferred step, and nothing in the assistants' crawler documentation promises a rendering step at all. Text that matters belongs in the HTML.

It does nothing about content kept in a file. 2.1% of the sites in the audit had their real substance, the price list or the service menu, inside a PDF. And it does nothing about the bot wall: for the 7.2% behind a challenge page, whatever robots.txt said was beside the point, because an assistant challenged at the front door never gets far enough to read it. A permission the front door contradicts is not a permission.

How to check your own site, in order

1. Open /robots.txt on your own domain and read which tokens have a group of their own. If none of the nineteen appears, the * group is what the crawlers among them read. The fetchers that act on a person's request are the exception: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally does not follow them, because a person asked for the page.
2. Run the crawler access check. It reads the file the way each of the nineteen tokens would, tells you which are allowed and which are blocked, and says whether the rules came from your file or were injected by your CDN.
3. Fetch one page as a crawler would and look at the status code, not the page: curl -A "OAI-SearchBot" -I https://www.example.com/. A 200 is a page; a 403, a 503 or an HTML challenge is a bot wall, and the setting that raises it lives in your CDN's bot management, not in robots.txt.
4. View the source of your service pages with scripts disabled. If the words you sell with are not in the HTML you are looking at, they are not in the HTML a fetcher receives.
5. Decide about the training-side tokens, write the decision into the file in its own group, and date the change somewhere you will find it. Then leave the file alone; the answer-side rules only ever need to change when an operator publishes a new token.

None of this makes an assistant name you. It removes the reasons an assistant could not. What it names once it can read the page is decided by what the page says, which is the subject of everything else on this site, starting with how we measure it.