AI search engines at a glance: who fetches your pages, and how each answers

What counts as an AI search engine here

Any product that answers a question by writing a reply, and that can fetch web pages to write it. That rules in the chat assistants when their web search is on, and Google's own answers on the results page. It rules out a plain list of links. The glossary calls the family answer engines; the difference from a search engine is that the page is read and summarized rather than listed.

Two ways of knowing coexist in every one of them. There is what the model learned in training, which stops at a date, and there is what it retrieves when a question sends it to the web. Only the second can be changed by editing a page this month, which is why the rest of this article is about the crawlers and fetchers that do the retrieving.

The six, side by side

Every row below comes from the operator's own crawler documentation or, for the last column, from what the product shows a user. Where a cell is blank, we found nothing documented and say nothing. The token names are the ones our crawler access check reads a robots.txt for.

The sources for each row: OpenAI's crawler documentation for the OpenAI tokens in the table (its page lists a fourth, OAI-AdsBot, which visits only pages submitted as ads) and the fact that OAI-SearchBot surfaces and links sites in ChatGPT's search; Anthropic's web search tool documentation for Claude citing what it retrieved; Google's list of common crawlers for Googlebot and Google-Extended, which governs Gemini training and grounding and has no effect on Search; Google's page on AI features for the supporting links; Perplexity's crawler documentation for its two tokens; and Bing's crawler page for Bingbot, which our crawler table lists as the crawler behind Bing search and Copilot answers.

What all six have in common

They read HTML. Every fetcher in the table receives the page your server sends, and none of the assistants' crawler documentation promises to run your scripts first. Google renders JavaScript, but its own documentation describes rendering as a separate, deferred step. Text that exists only after a script runs is text most of these products may never see.

They obey robots.txt for their indexing crawlers, token by token, under RFC 9309. A group written for one token binds that token and no other. The user-triggered fetchers are the exception Perplexity states plainly: Perplexity-User generally does not consult the file, because a person asked for the page.

And they are all stopped by the same things at the front door. In our audit of 9,483 small business websites (40 trades, 90 metros, September 2026), 7.2% served our crawler a bot challenge instead of the site and 3.0% served an empty shell that only a script could fill. A wall like that is not addressed to any one product. It turns all six away together.

Where they differ, and why it matters to a business

They keep separate indexes. A page ChatGPT's crawler has read may not yet be in Perplexity's index, and the other way round. So a business can be named by one assistant and missing from another for reasons that have nothing to do with its pages: one crawler has been back since the last edit and another has not. That is why a measurement that asks one assistant tells you about that assistant only. The probe in our audit used Claude, and we report its 14.9% named rate as a Claude number for that reason.

They split training from answering differently. OpenAI, Anthropic and Google each separate the crawler that collects training data from the one that serves answers, and the robots.txt rule you write for one leaves the other alone. Disallowing GPTBot does not remove a site from ChatGPT's search answers, OpenAI's documentation says; disallowing OAI-SearchBot does. Disallowing Google-Extended does not remove a site from Google Search or its AI Overviews.

They show the buyer different things. Perplexity numbers its sources on every answer. Google's AI Overviews put supporting links beside the text. ChatGPT and Claude link or cite what they searched when a search ran. A buyer who reads a sourced answer can click through to you; one who reads an unsourced reply only has your name. Both are worth being in, and only the first sends a visit.

What to do with the differences

1. Allow every answer-side token in the table (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, Bingbot), and decide about the training-side tokens separately, in their own robots.txt group. Our robots.txt article has a file to copy.
2. Fetch your homepage with each answer-side user agent and read the status code. A challenge page for one of them is usually a challenge page for all of them, and the setting is in your CDN or security plugin.
3. Write the page once, for people. Google's documentation says its AI features need no special markup or file, and none of the other operators documents a separate format. One set of pages with the facts in text serves all six.
4. Measure each product separately with the same questions. A business named by Perplexity and absent from ChatGPT has a freshness or access gap, not a content gap, and the fix is different.
5. Record which sources each answer shows. The pages an assistant cites instead of yours are the pages to read next.

What this comparison leaves out

How any of them ranks or weighs one page against another. No operator documents it, we have not measured it, and we do not guess. The table describes access and display, which are documented and checkable. What each product does with the pages once it has them is the part a business can only observe from the outside, by asking the same questions over time and reading the answers. Our measurement does that for one of the six, Claude, and does not report Claude's answer as any other product's.