Search Engine Crawler
A search engine crawler is the program a search engine runs to fetch web pages, follow the links on them and hand what it finds to an index. Googlebot is the best known. Every engine, and every AI assistant that retrieves pages before answering, runs one or more of its own.
What it does with a page
Google describes the pipeline in three stages: crawling, in which URLs are discovered from links and sitemaps and fetched; indexing, in which the page is analyzed, checked for duplication and stored; and serving, in which the index is queried. Googlebot renders pages in a recent version of Chrome, so content produced by JavaScript is seen. It reads robots.txt before fetching and stays out of disallowed paths.
How an AI assistant’s crawler differs
In two ways that matter to a site owner. First, one company runs several agents with separate jobs: OpenAI documents OAI-SearchBot for search, GPTBot for training data and ChatGPT-User for fetches a user asked for; Perplexity documents PerplexityBot and Perplexity-User. A rule that fits Googlebot fits none of them by default. Second, a retrieval fetch happens at answer time, on the user’s behalf, and the page fetched a moment ago is what the answer is written from; Perplexity says its user-initiated fetcher generally ignores robots.txt for that reason. Many of these fetchers do not execute JavaScript.
This site’s own crawler, identifying itself as gaash-audit/1.0 and running no JavaScript, fetched 9,483 small-business sites. 3.0% rendered their content only in JavaScript, so a crawler that did not run scripts saw an empty page, and 7.2% served a bot-protection challenge instead of the site. Both are in what 9,483 small business websites serve to an AI crawler.
A common question
How does an assistant's fetcher differ from Googlebot when it reads a site?
In two ways that change what a site owner has to do. Googlebot renders pages in a recent version of Chrome, so content produced by JavaScript is seen, while many retrieval fetchers execute no scripts at all; this site's own crawler ran that way across 9,483 small-business sites and found 3.0% handing back an empty page as a result. Separately, one company runs several agents with their own tokens and jobs, so a rule written for Googlebot addresses none of them by default.
Related terms
Sources
Pipeline and rendering: How Google Search works. AI agents: OpenAI and Perplexity crawler documentation. The audit figures are this site’s own; sample drawn per metro and trade, percentages describe the sample.