Indexing
Indexing is the step where a search engine (or an AI system with its own retrieval index) takes a page it has already crawled and adds it to the searchable dataset it draws answers from. Crawling gets a page seen; indexing is what gets it stored, and only stored pages can surface in results, in citations, or in an AI-generated answer.
Why indexing matters
A page can be well-written, fast, and well-linked and still be functionally invisible if it never makes it into the index. For you this means indexing is the floor, not a ranking factor you optimize on top of — it's the prerequisite that everything else depends on. This matters just as much for AI visibility as for classic search: Google's AI Overviews now reach over 2 billion monthly users across 200+ countries, and tools like ChatGPT and Gemini increasingly retrieve live web results rather than answering purely from training data. If a page isn't indexed, it can't be pulled into any of those answers, no matter how well the content itself is written.
How indexing works
There are three distinct stages. Crawling comes first: a bot requests the page and reads its content. Processing follows: the content is parsed, rendered, and evaluated. Indexing is the last step — actual admission into the searchable database. You influence this with robots.txt (which controls what crawlers may fetch), the noindex meta tag (which blocks admission even after a successful crawl), and an XML sitemap that tells crawlers which URLs matter. A canonical tag helps consolidate near-duplicate pages so only one version gets indexed. The key distinction to hold onto: crawling and indexing are separate events. A page can be crawled repeatedly and still never be indexed.
Common mistakes
The most common failure is an accidental noindex tag left over from staging or a CMS default. Almost as common is a robots.txt rule that blocks the crawler outright, so the content is never even read. Thin or near-duplicate pages are often deliberately excluded because they add nothing distinct to the index. Broken redirect chains, wrong HTTP status codes, and weak internal linking also leave pages stranded. One mistake worth naming directly: publishing an llms.txt file does not fix any of this. Google has said no Search system reads or acts on llms.txt, and an Ahrefs analysis of roughly 137,000 sites that published one found about 97% saw no measurable referral traffic tied to it. If a page is missing from the index, check your search console and fix the actual crawling or indexing signal — don't treat a new file format as a substitute.
Relation to AI visibility
For GEO, indexing is a precondition, not a lever you pull for AI specifically. Many AI assistants and generative search products answer live queries by retrieving from a search index and summarizing what they find; an unindexed page simply cannot be selected as a source. Some providers also run their own crawlers (like GPTBot) that build separate retrieval indexes for their models. Worth knowing: Ahrefs research shows only about 6–8% of URLs cited by ChatGPT overlap with Google's top 10 for the same query, so being indexed and ranking well in classic search doesn't guarantee AI citation — but being indexed somewhere is still the non-negotiable first gate. Google is also explicit that no special schema, markup, or AI-specific file is required to be eligible for AI Overviews or AI Mode; the underlying requirement is still ordinary crawlability and indexing.
Example
Imagine a small tax firm in Leipzig that publishes a detailed new guide on deducting home-office costs. The page looks good and reads well, but weeks later it hasn't brought a single visitor from search. The cause: during a site redesign, a noindex tag from the staging environment was accidentally left in the live page's head. The crawler visited the page and read it fine — it just never admitted it into the index. Once the firm's developer removes the tag and requests indexing through the search console, the page starts appearing in results within days.
Common questions
How long does indexing a new page take?
It varies widely, from hours to several weeks. A clean XML sitemap, solid internal linking, and manually requesting indexing through the search console all tend to speed it up, but none of them guarantee a specific timeline.
What's the difference between crawling and indexing?
Crawling is a bot visiting and reading your page. Indexing is the separate, later step of admitting that content into the searchable database. A page can be crawled many times and still never get indexed.