Crawling

Q: What is the difference between crawling and indexing?

Crawling is the reading of a page by a bot. Indexing is the next step: the captured content is added to the search engine's database so that it can appear in results. Only what has been crawled can be indexed – but not everything crawled automatically ends up in the index.

Q: Can I control which crawlers visit my site?

Yes. Via the robots.txt file you define which bots may enter which areas. You can specifically allow or exclude individual crawlers such as GPTBot. But note: robots.txt is an instruction, not a technical enforcement. Reputable crawlers abide by it, but exclusion cannot be guaranteed this way.

Crawling is the process by which an automated program — a bot, also called a crawler or spider — fetches a web page, reads its HTML, and follows the links it finds to discover more pages. Search engines like Google run crawlers, and so do the companies behind AI systems: OpenAI's GPTBot, Google's Google-Extended, and similar bots fetch pages to build the training data and live-retrieval indexes that power search results and AI-generated answers. If a page is never crawled, it cannot be indexed and cannot be cited — it simply does not exist as far as any of these systems are concerned.

Why crawling matters

Crawling is the gate everything else passes through. A page can have excellent content, but if no crawler ever reaches it, that content is invisible to Google, to ChatGPT, to Perplexity, and to every other system that answers questions using web content. This matters more, not less, in an AI-search world: Google's AI Overviews now reach more than 2 billion monthly users across 200+ countries, ChatGPT has around 900 million weekly active users, and Gemini has passed 1 billion monthly active users. All of that traffic is fed, at some point, by crawlers reading pages. Getting crawled is a precondition for visibility, not a guarantee of it — but without it, nothing downstream is possible.

How crawling works

A crawler starts from a set of known URLs, fetches each one, and parses the response. Links it finds get added to a queue, so it works outward from page to page. Crawlers check robots.txt first to see which paths they're allowed to fetch, and an XML sitemap gives them a more reliable list of pages to prioritize. How much of a site actually gets crawled, and how often, depends on crawl budget — the finite amount of time and requests a bot is willing to spend on a given domain. Fast response times, a sane URL structure, and solid internal linking all make that budget go further, so more of the site actually gets seen.

Common mistakes

Most crawlability failures are self-inflicted. A stray Disallow rule in robots.txt can lock a bot out of an entire folder without anyone noticing. Content that only renders after client-side JavaScript runs is invisible to crawlers that don't execute scripts, or that time out before they finish. Redirect chains, infinite parameter combinations, and 5xx errors all burn through crawl budget on pages that don't matter, leaving the ones that do unfetched. A page with no internal links pointing to it is also easy to miss entirely, no matter how good it is. One mistake worth naming directly: publishing an llms.txt file and assuming it helps. Google's John Mueller has confirmed no Google Search system reads or acts on llms.txt, and an Ahrefs analysis of roughly 137,000 sites that published one found about 97% saw zero measurable referral traffic tied to it. Google's own guidance is explicit that no special file, markup, or schema is required for AI Overviews or AI Mode, and warns against writing content specifically "for AI" instead of for the crawlers and readers who already visit the site.

Relevance to AI recommendations

AI systems don't skip crawling — they depend on it, sometimes through more than one bot. GPTBot collects material OpenAI may use for training, while separate retrieval crawlers fetch pages at query time to ground a live answer. Blocking these bots in robots.txt removes you from consideration entirely; allowing them is necessary but not sufficient. What a bot fetches once it arrives still has to be worth citing, and that selection process doesn't mirror classic search rankings: an Ahrefs study found only about 6–8% of URLs ChatGPT cites overlap with Google's top 10 for the same query, and roughly 80% of ChatGPT-cited URLs don't rank in Google's top 100 at all. Crawling gets you in the room; it doesn't decide whether you get quoted.

Example

Imagine a small tax firm in Leipzig that publishes a weekly explainer on tax-filing rules. The partners can't understand why none of it shows up in Google, let alone in an AI answer. Checking robots.txt turns up the problem: a rule meant to block an old staging folder is accidentally blocking the entire blog. Googlebot has never been able to read a single post. Once the rule is fixed and a current XML sitemap is submitted, Google crawls and indexes the backlog within days, and the posts start appearing in search results shortly after.

Common questions

What is the difference between crawling and indexing?

Crawling is a bot fetching and reading a page. Indexing happens after: the content is processed and stored so it can actually surface in results. A page has to be crawled before it can be indexed, but crawling a page is no guarantee it gets indexed — low-value or duplicate pages are often crawled and then discarded.

Can I control which crawlers visit my site?

Partially. robots.txt lets you name specific bots, including AI crawlers like GPTBot, and tell them which paths they may or may not fetch. That said, robots.txt is a request, not an enforcement mechanism. Well-behaved crawlers honor it; nothing stops a crawler that chooses to ignore it, so it's a signal, not a lock.