Retrieval-Augmented Generation (RAG)

Retrieval-augmented generation is the method by which a language model is given documents fetched for the current question and writes its answer from them rather than from memory alone. Retrieval happens at answer time, so the answer can contain facts newer than the model’s training and can cite where each came from.

The three steps

Retrieve, insert, generate. Anthropic’s documentation of Claude’s web search tool shows the shape: the model decides a question needs current information and issues a search, the results are provided back to it, the search can repeat within one request, and the final answer carries citations. Each citation records the URL, the title and the cited text, which the documentation caps at 150 characters. A citation is a span, not a page.

What it means for a website

Every request re-reads the page. Whether it is retrieved at all depends on the search engine’s ranking and on whether the fetcher gets HTML back: in this site’s own audit of 9,483 small-business sites, 7.2% served a bot-protection challenge instead of the page and 3.0% rendered their content only in JavaScript, so a fetcher that ran no scripts got nothing. Whether it is cited depends on a passage answering the question within a short span. Whether the business is named depends on the pages agreeing about who it is. The audit has the figures.

RAG and training data

The two are separate channels. OpenAI’s crawler documentation gives them different tokens: GPTBot collects pages for training, OAI-SearchBot indexes pages for ChatGPT search, and opting out of one does not opt out of the other. A site that blocks training can still be retrieved and cited; a site that blocks the search crawler drops out of retrieval however much the model learned from it.

A common question

What has to be true of a page for it to be retrieved and then cited?

Three separate conditions, tested on every request, because each request re-reads the page. Retrieval depends on the search engine's ranking and on the fetcher getting HTML back at all; in the audit of 9,483 small-business sites, 7.2% returned a bot-protection challenge and 3.0% put their content behind JavaScript, so a fetcher running no scripts came away with nothing. Citation depends on a passage answering the question inside a short span, since a citation records a span of text rather than a whole page. Being named depends on the pages agreeing about who the business is.

Sources

Search, insertion and citation shape: Anthropic, web search tool. Crawler tokens: OpenAI, crawlers. The audit figures are this site’s own; sample drawn per metro and trade, percentages describe the sample.