Training Data
Training data is the body of text, and for some models images and audio, that a machine-learning model is fitted to before anyone uses it; whatever the model knows without looking anything up came from there. For a large language model it is a corpus of web pages, books, code and licensed material, frozen at a cutoff date.
Training data and retrieval are different channels
What a model learned in training is fixed in its weights and stops at the cutoff. An assistant answering a question about a local business today does not rely on that; it searches, fetches pages and writes from what it just read. So a business can be absent from every training set and still be named, because the assistant found its site an hour ago, and a business that appears in the corpus can go unnamed because the page it fetched today said nothing usable. Being recommended is a retrieval outcome far more than a training one.
Opting out, and what it does not do
Crawlers that collect training data identify themselves and can be refused in robots.txt under RFC 9309. OpenAI’s crawler documentation keeps the jobs apart: GPTBot gathers material that may be used for training, OAI-SearchBot indexes for ChatGPT search, and ChatGPT-User fetches on a user’s behalf. Disallowing GPTBot keeps a site out of future training sets. It does not remove the site from ChatGPT search answers; disallowing OAI-SearchBot does that. A site owner deciding on one token is deciding one thing.
In practice few sites refuse either. Of 9,483 small-business sites this site audited, 3.2% disallowed AI crawlers in robots.txt, and more than twice that share were behind bot protection that blocked them without anyone deciding to. Both figures are in what 9,483 small business websites serve to an AI crawler.
A common question
Does blocking a training crawler keep a site out of an assistant's search answers?
No, and the two jobs are worth keeping apart. OpenAI's documentation separates them: GPTBot gathers material that may be used for training, OAI-SearchBot indexes for ChatGPT search, and ChatGPT-User fetches on a person's behalf. Refusing GPTBot keeps a site out of future training sets and leaves search answers untouched; refusing OAI-SearchBot is what removes it from those. A site owner ruling on one token is ruling on one thing, so the choice needs to be made per token.
Related terms
Sources
Robots rules: RFC 9309. Crawler roles: OpenAI’s crawler documentation. The robots figure is this site’s own, computed from its audit records; sample drawn per metro and trade.