GPTBot
GPTBot is the user-agent token sent by the crawler OpenAI uses to fetch web pages for training its models. It is one of three OpenAI tokens, and the one most often blocked for a reason that has nothing to do with appearing in ChatGPT’s answers.
Three tokens, three jobs
OpenAI’s crawler documentation lists them. GPTBot collects pages for training foundation models; disallowing it says the site’s content should not be used that way. OAI-SearchBot indexes pages so they can be surfaced in ChatGPT search; disallowing it removes the site from those answers. ChatGPT-User fetches a page when a person asks ChatGPT something that needs it, or uses a custom GPT; because a person triggered the request, OpenAI says robots.txt rules may not apply to it. All three publish their IP ranges, so a server log can tell them apart from something pretending to be them.
The practical consequence is the one owners get backwards. Blocking GPTBot keeps a site out of the next training run. It does not keep the site out of ChatGPT search, and it does not stop a user-initiated fetch. A site that wants to be recommended and not trained on allows OAI-SearchBot and disallows GPTBot; robots.txt reads each group by its own product token, under the longest-match rule in RFC 9309, so the two can be set independently.
What small-business sites do
Mostly nothing, and sometimes the wrong thing by accident. Across 9,483 small-business sites this site audited in September 2026, 3.2% disallowed AI crawlers in robots.txt. A larger share, 7.2%, sat behind bot protection that served the crawler a challenge page instead of the site, which shuts out every assistant at once without anyone having decided to. The figures are this site’s own audit and are in what 9,483 small business websites serve to an AI crawler.
A common question
Does blocking GPTBot keep a business out of ChatGPT's answers to customers?
No, and owners commonly expect the opposite. GPTBot gathers pages for training foundation models, so disallowing it keeps the site out of the next training run and nothing else. ChatGPT search runs on the index that OAI-SearchBot builds, and a page a person asks about can still be fetched live by ChatGPT-User. A site that wants to be recommended without being trained on allows OAI-SearchBot and disallows GPTBot, because robots.txt reads each group by its own product token.
Related terms
Sources
Tokens and their effects: OpenAI, crawler documentation. Rule matching: RFC 9309. The 3.2% and 7.2% figures are this site’s own; sample drawn per metro and trade, percentages describe the sample.