AI crawler

An AI crawler is an automated client that an AI company runs to fetch web pages, announcing itself with a product token in its user agent. What it fetches goes to one of three places: a training corpus, a search index, or the answer being written for one person right now.

The tokens and what each one does

Each company runs more than one, and the token tells you the job. OpenAI’s crawler documentation lists GPTBot, which collects text that may be used to train models; OAI-SearchBot, which indexes sites for ChatGPT search; and ChatGPT-User, which fetches a page when a person’s request calls for it. Disallowing GPTBot says nothing about search; disallowing OAI-SearchBot removes a site from ChatGPT search answers. Perplexity’s crawler documentation draws the same line between PerplexityBot, which indexes, and Perplexity-User, which fetches on a person’s behalf and, in the company’s own words, generally ignores robots.txt because a person asked. Google’s list of common crawlers adds a third pattern: Googlebot feeds Search, and a separate Google-Extended token controls whether crawled content trains or grounds Gemini, with no effect on inclusion in Search.

What a crawler is actually served

Not always the page. This site’s crawler, identifying itself as gaash-audit/1.0 with JavaScript off, fetched the homepage, contact page and about page of 11,060 small-business sites in September 2026; 9,483 answered. Of those, 3.2% disallowed AI crawlers in robots.txt. A further 7.2% sat behind bot protection that returned a challenge page instead of the site, 3.0% rendered their content only in JavaScript, and 2.1% kept the substance in a PDF. Each of those is a page an assistant cannot read, and only the first was chosen. The breakdown is in what 9,483 small business websites serve to an AI crawler.

AI crawler and robots.txt

The file is read the way RFC 9309 says: rules are grouped by product token, the most specific matching path wins, and where an allow and a disallow are equally specific the allow wins. A group addressed to one token does not bind another, so a rule written for GPTBot alone leaves OAI-SearchBot, PerplexityBot and Googlebot where they were. And the user-triggered fetchers may not consult the file at all.

A common question

Does blocking GPTBot in robots.txt keep a site out of ChatGPT search results?

No, because each company runs more than one crawler and the product token says which job a fetch belongs to, so a rule written for the token that collects training text leaves the search indexer exactly where it was. Rules in the file are grouped by token, and a group addressed to one does not bind another. The same split shows up elsewhere: one Perplexity token indexes while another fetches on a person's behalf, and Google keeps its search crawler apart from the token governing training and grounding. Removing a site from ChatGPT search means disallowing OAI-SearchBot instead.

Sources

Token roles: OpenAI’s crawler documentation, Perplexity’s crawler documentation, Google’s list of common crawlers. Parsing rules: RFC 9309. The reachability figures are this site’s own, computed from its audit records; sample drawn per metro and trade, not from a random frame.