How ChatGPT and Gemini get to a driving school's pages

Which OpenAI crawlers fetch a driving school's pages?

OpenAI's crawler documentation names four user agents. The fourth, OAI-AdsBot, visits only pages submitted as ads; each AI crawler behind the other three has one job that touches a driving school's site. OAI-SearchBot indexes pages for ChatGPT search and the citations in its answers. ChatGPT-User fetches a page when a ChatGPT user asks about it, as a parent might by pasting your prices page into the chat and asking what the package includes. GPTBot collects training data for OpenAI's models.

The same document states the consequence. Disallowing OAI-SearchBot removes a site from ChatGPT's search answers; disallowing GPTBot does not. A school that blocked GPTBot because it did not want its lesson descriptions in a training set made a decision about training, not about being named. A school that blocked OAI-SearchBot took its packages, prices and pickup area out of the answers ChatGPT search can cite.

Which Google tokens stand behind Gemini and AI Overviews?

Google's common crawlers documentation separates two things a school can easily run together. Googlebot is the crawler that feeds Google Search, and AI Overviews and AI Mode are part of Search. Google-Extended is a token that controls whether content Google has crawled may be used for Gemini's training and grounding, and it has no effect on whether a page appears in Search.

For a school that means two separate decisions. Disallowing Googlebot stops Google from reading the prices page, which leaves it nothing to build a snippet from, in AI Overviews or anywhere else in Search. Disallowing Google-Extended leaves Search alone and tells Google not to use the page for Gemini's training or its grounding, the step in which an answer is tied to pages. A school that wants its packages within reach of a Gemini answer grounded in the web leaves Google-Extended open.

Gemini, the assistant, and AI Overviews, in Search, are different products under different controls, and a school can appear in one without the other. Measure them apart. Our method does not measure Gemini: it puts every question to one assistant, Claude, so a reading of Gemini is one you take yourself, in Gemini.

What Google says its AI features need from a page

Google's page on AI features and your website is short on requirements. There is no special markup and no special file for AI Overviews or AI Mode. To be shown as a supporting link, a page has to be indexed and eligible to appear with a snippet. Beyond that, it points back to the fundamentals of SEO and to content written for people.

Read that against a school's own pages. A prices page that still carries a noindex tag from a redesign cannot be a supporting link, because it is not in the index at all; Google's noindex documentation adds that a page blocked in robots.txt cannot even have its noindex read. That is indexing failing before a word on the page matters.

A course timetable published only as a picture of a flyer has a different problem. Google's snippet documentation names the page text and the meta description as sources a snippet is built from, and a picture supplies neither. The dates of next month's intensive course belong in the page as text.

None of these documents says how Google chooses between two schools that both qualify, and this article does not claim to know. They describe access and eligibility. What an answer then says about a school is outside them.

What the crawlers receive from a school's site

A crawler gets the HTML the server sends. Google also renders scripts, but its JavaScript documentation describes rendering as a separate step that comes later, and OpenAI's crawler documentation says nothing about running scripts at all. So what counts is what is in the HTML. Our audit of 9,483 small business websites measured four ways a site can fail to hand it over, across 40 trades; driving schools were among them, and we publish no cut by trade.

The first row is the one nobody chose. Bot protection acts before robots.txt is ever read, so a crawler that is challenged gets no further than the challenge. The third can come from the tools a school uses to take bookings: if the calendar is embedded from a booking service and drawn by a script, the HTML a crawler receives may hold an empty frame where the times and prices should be.

For the last three rows the fix is the same in kind: the facts a learner and a parent ask about, stated as text in the page. The prices page article shows the shape, and the PDF trap covers files in general. For the first, the fix is a setting in your CDN or security plugin; ask whoever manages it which crawlers it lets through, by name.

How to check what ChatGPT's and Google's crawlers may read

1. Open /robots.txt on your own domain and look for groups naming OAI-SearchBot, ChatGPT-User, GPTBot, Googlebot or Google-Extended. Under RFC 9309 a token with its own group follows only that group, and a group written for one token does not bind another.
2. Run the crawler access check. It reads your robots.txt the way each of 19 AI crawler tokens would and reports which are allowed in and which are asked to stay out.
3. Fetch the prices page without a browser and read the status code: curl -I https://www.example.com/prices. A 200 means a page came back. A 403, a 503 or a page asking you to prove you are human means bot protection stands in the way, and since some protection treats a real crawler differently from your laptop, ask whoever runs it what it does with each token.
4. View the source of the prices, packages and booking pages with scripts switched off, and search it for a price. If the number is not in the source, a fetcher that runs no script does not have it either.
5. Search the source of every page you want in Google's AI features for noindex. A page that asks to stay out of the index cannot be a supporting link, whatever it says.
6. Make the two training decisions on purpose. GPTBot affects OpenAI's training and not ChatGPT's search answers. Google-Extended affects Gemini's grounding as well as its training, so blocking it has a cost a school may not intend.

The robots.txt article has the full token table and a file you can copy. What a school's pages should say once the crawlers can reach them is laid out in the overview for driving schools.