OpenAI
OpenAI is the company that makes ChatGPT and the GPT family of models, and one of the operators whose crawlers determine whether a website can appear in an assistant’s answers. To a site owner it is not one visitor but several user agents with different jobs.
The crawlers and what each one is for
OpenAI’s crawler documentation names them. OAI-SearchBot is used to surface websites in ChatGPT’s search features; a site that disallows it will not appear in ChatGPT search answers, though it may still show as a navigational link. GPTBot crawls content that may be used to train OpenAI’s models; disallowing it signals that the site’s content should not be used for training. ChatGPT-User handles actions a user initiates, such as visiting a page during a conversation; it is not used for automatic crawling, and the documentation notes that robots.txt rules may not apply to requests a user triggers. There is also OAI-AdsBot, which checks pages submitted as advertisements.
What one robots.txt line does
Under RFC 9309, a robots.txt file is read in groups addressed to product tokens, and the rule that applies is the one whose path matches longest. A group for GPTBot says nothing to OAI-SearchBot. So a site that blocked GPTBot to stay out of training data has made no decision about whether ChatGPT can cite it, and a site that wanted to be cited but blocked every unfamiliar bot has made the opposite decision by accident. In this site’s audit of 9,483 small-business sites, 3.2% disallowed AI crawlers in robots.txt and 7.2% sat behind bot protection that served a challenge page in place of the site; the second group blocks assistants without anyone having chosen to. Both figures are in what 9,483 small business websites serve to an AI crawler.
What this site has not measured
Whether ChatGPT names any given business. The one-question probe behind the audit was run against Claude, and this site does not present a Claude number as a ChatGPT rate. The visibility check puts its panel of questions to one assistant, Claude, and reports a Claude reading, not a ChatGPT one.
A common question
Does blocking GPTBot also keep a site out of ChatGPT's search answers?
No, because a robots.txt file is read as groups addressed to product tokens, with the longest matching path deciding which rule applies, so a group written for GPTBot says nothing at all to OAI-SearchBot. Blocking GPTBot signals that the content should not be used for training and settles nothing about whether ChatGPT can cite the site. The reverse trap is a site that wanted citations and blocked every unfamiliar agent, making the opposite choice by accident. In the audit, 3.2% of 9,483 sites disallowed AI crawlers and 7.2% sat behind bot protection.
Related terms
Sources
Crawler roles: OpenAI’s crawler documentation. Robots rules: RFC 9309. The robots.txt and bot-protection figures are this site’s own, from its audit records; sample drawn per metro and trade.