Indexing

Indexing is the stage where a search engine analyzes a page it has crawled and stores what it learned, so that the page can be a candidate the next time someone searches. A page that was fetched but never indexed cannot appear in a result, however good it is.

Between the fetch and the index

Google describes its pipeline as crawling, indexing and serving. In the middle stage it reads the text, the title element and the alt attributes, renders the page with a recent Chrome so that content drawn by JavaScript is seen, then works out which other pages are duplicates of this one and picks a canonical to represent the group. It also records the page’s language, the country it is relevant to and how usable it is. The same document says plainly that indexing is not guaranteed: a page can be crawled, processed and left out.

Keeping a page out on purpose

A noindex rule, either as a robots meta element or an X-Robots-Tag response header, asks the engine to drop the page. It only works if the crawler can fetch the page: a URL blocked in robots.txt is never read, so the rule is never seen, and the URL can still be indexed from links elsewhere. Removal waits for the next visit, which for an unimportant page can be months away.

Two roads into an assistant’s answer

An AI assistant that searches the web relies on an index too, though not always its own. OpenAI documents that OAI-SearchBot builds the index behind ChatGPT search, and that opting out of it removes a site from those answers, while ChatGPT-User fetches a page live when a person asks about it. So a page reaches an answer either because it was indexed beforehand or because it was fetched on the spot, and each road has its own crawler token and its own robots rule.

A common question

Why does a noindex rule fail when the URL is also blocked in robots.txt?

The rule travels with the page, so a crawler has to fetch the page to see it. A URL disallowed in robots.txt is never read, the rule is never encountered, and the URL can still be indexed from links pointing at it elsewhere. Even where the rule is seen, removal waits for the next visit, which for a page the engine treats as unimportant can be months away. Blocking a fetch and asking for removal are two different instructions, and only one of them needs access.

Sources

Google Search Central, in-depth guide to how Google Search works and block search indexing with noindex. OpenAI, crawler documentation.