Technical SEO

Technical SEO is the part of search optimization concerned with whether a crawler can reach, read, render and index a site at all: server responses, robots rules, redirects, rendering, sitemaps, canonical addresses and load time. It decides whether a page is evaluated, not how it scores once it is.

The checks that also gate an AI assistant

An assistant with web retrieval fetches pages the way a search crawler does, so the same failures stop it. Robots rules are read per product token under RFC 9309, and a disallow aimed at one bot is a disallow for that bot only. Bot-protection services can answer a fetch with a challenge page instead of the site. Content assembled in the browser by JavaScript is invisible to a fetcher that does not execute scripts; Google’s JavaScript documentation explains that its own crawler renders in a separate, later step, and other fetchers may not render at all. A price list kept in a PDF is not on the page.

These are small numbers and they add up. Across 9,483 small-business sites this site audited with a non-rendering crawler, 3.2% disallowed AI crawlers in robots.txt, 7.2% served a bot-protection challenge instead of the site, 3.0% rendered their content only in JavaScript, and 2.1% kept their real content in a PDF. Each of those sites had, for the purposes of an assistant, no content. The breakdown is in what 9,483 small business websites serve to an AI crawler.

What a clean technical audit does not tell you

Passing every check means the page arrives. It says nothing about what is on it. A site with a perfect robots file, fast responses and a valid sitemap can still contain no passage an assistant can quote and no consistent statement of who the business is. Technical SEO is the gate. What is behind the gate is content and identity, which are separate work.

A common question

If a site passes every technical check, is it ready to be quoted by an assistant?

No. Passing the checks means the page arrives, which says nothing about what is on it. A site with a clean robots file, fast responses and a valid sitemap can still hold no passage an assistant could quote and no consistent statement of who the business is. The technical work is the gate, and what sits behind the gate is content and identity, which are separate jobs.

Sources

Robots rules: RFC 9309. Rendering: Google’s JavaScript SEO basics. The reachability figures are this site’s own, computed from its audit records with a crawler that did not execute JavaScript; sample drawn per metro and trade.