HTTP Status Code

An HTTP status code is the three-digit number a server sends at the top of every response, telling the client whether the request succeeded, was redirected, failed because of the request or failed because of the server. A crawler reads it before it reads anything else.

The five classes

RFC 9110, section 15 defines them by first digit. 1xx is informational: the request was received and work continues. 2xx means the request succeeded. 3xx means the client has to do something more, usually follow a redirect. 4xx means the request itself was wrong or cannot be met, and 5xx means the server failed on a request that looked valid. A client that meets a code it does not know is expected to treat it as its class, so an unfamiliar 4xx is still an error at the client’s end.

What a search crawler does with each

Google documents its handling by class. A 2xx response is considered for indexing but not guaranteed it. A 301 or 308 is a strong signal to process the target instead; a 302 or 307 is a weaker one; up to ten hops are followed and only the final page counts. Any 4xx means the content does not exist: it is ignored, indexed copies are removed over time and crawling slows, with 404 and 410 treated alike. The exception is 429, which Google reads as an overloaded server rather than a missing page. A 5xx cuts the crawl rate in proportion to how many URLs return it, and pages that keep failing eventually drop out. A 2xx that carries an error page or an empty page is flagged as a soft 404.

The code is not the page

A status code says what kind of response arrived, not whether the body is the site. Across 9,483 small-business sites audited by this site in September 2026, 7.2% sat behind bot protection that served a challenge page in place of the site, and 3.0% rendered their content only in JavaScript, so a crawler that does not run scripts received a page with nothing on it. Whatever code accompanies such a response, that page is what the assistant has to work from. The breakdown is in what 9,483 small business websites serve to an AI crawler.

A common question

Does a 200 response mean an assistant actually received the content of a page?

Not by itself. A status code reports what kind of response arrived, not what the body held, and a 2xx carrying an empty or error page is flagged as a soft 404. In the audit of 9,483 small-business sites, 7.2% sat behind bot protection that returned a challenge page in place of the site, and 3.0% rendered their content only in JavaScript, leaving a crawler that runs no scripts with nothing. Whatever code came with such a response, that page is what the assistant works from.

Sources

Status code classes: RFC 9110, HTTP Semantics, section 15. Crawler handling: Google Search Central, how HTTP status codes and network errors affect Google Search. The audit figures are this site’s own; sample drawn per metro and trade, percentages describe the sample.