Crawling

Crawling is the automated fetching of web pages by a program that follows links, reads what each page serves and records it for later use. Search engines crawl to build an index; AI assistants crawl, or fetch on demand, to find text they can answer from. What the program receives is the only version of the site it knows.

What robots.txt decides

Before fetching, a well-behaved crawler reads /robots.txt and applies the rules for its own product token. RFC 9309 sets the mechanics: rules are grouped by token, a crawler with no group of its own uses the wildcard group, the rule with the longest matching path wins, and where an allow and a disallow are equally specific the allow is taken. Two failure modes are easy to miss. If robots.txt itself returns a client error, the crawler may treat the whole site as open; if it returns a server error, the crawler is to assume the whole site is closed until the file can be read again.

Search crawlers and assistant fetchers

Googlebot crawls, queues the page for rendering in a current Chromium, then indexes the rendered result, as Google’s JavaScript SEO basics describes; content built by JavaScript reaches its index late but does reach it. Assistant fetchers are typically lighter. Some are indexers that honor robots.txt, like OAI-SearchBot and PerplexityBot. Others fetch a single page because a user asked for it, and both OpenAI’s crawler documentation and Perplexity’s crawler documentation say those on-demand agents may not apply robots.txt rules; Google’s page on user-triggered fetchers says the same of its own. A site that relies on robots.txt to keep assistants out is therefore controlling the indexers and not the in-conversation fetches.

What 9,483 sites served

This site’s own crawler, identifying itself as gaash-audit/1.0 and executing no JavaScript, fetched the homepage, contact page and about page of 11,060 small-business sites; 9,483 answered. Of those, 3.2% disallowed AI crawlers in robots.txt, 7.2% were behind bot protection that returned a challenge page instead of the site, and 3.0% rendered their content only in JavaScript, so the crawler received an empty page. The full results are in what 9,483 small business websites serve to an AI crawler; the crawler access check runs the robots.txt part of that fetch against one domain.

A common question

Can robots.txt stop an assistant from fetching a page during a conversation?

Not reliably. The indexing fetchers, such as OAI-SearchBot and PerplexityBot, do read the file and follow the group that matches their token. The agents that fetch one page because a person asked for it are described differently by their own operators: OpenAI, Perplexity and Google all state that a user-triggered fetch may not apply the rules in robots.txt. So a site relying on that file to keep assistants away is governing the indexers while the in-conversation fetches carry on.

Sources

Robots rules: RFC 9309. Rendering pipeline: Google’s JavaScript SEO basics. On-demand fetchers and robots.txt: OpenAI’s crawler documentation, Perplexity’s crawler documentation, Google’s page on user-triggered fetchers. Audit figures are this site’s own; sample drawn per metro and trade, percentages describe the sample.