Search Engine Crawler
A search engine crawler is an automated program that requests web pages, reads their HTML and follows the links it finds so an index can be built. Googlebot and Bingbot are the classic examples; GPTBot and other AI crawlers now do a related job for language models. If a crawler can't reach a page, nothing downstream happens: it can't be indexed, ranked, or cited.
Why it matters
A page a crawler never reaches is a page that doesn't exist for search, full stop. That visit is the first link in a chain: crawl, then index, then rank or get cited. For traditional SEO this has always been table stakes. It matters just as much now that AI systems pull from their own crawls: whether ChatGPT, Perplexity, or Google AI Overviews can mention your brand often traces back to whether their bot was allowed in and could parse the page in the first place. Note, though, that being crawlable is a floor, not a strategy — Google has been explicit that there's no special markup or AI-only content that buys extra visibility once a page is reachable.
How it works
A crawler starts from a set of known URLs and works outward: it fetches a page, parses the source, extracts links, and queues those links for a later visit. It follows the rules it's given along the way — robots.txt marks off which areas it may enter, and an XML sitemap points it toward pages worth prioritizing. Every site gets a limited crawl budget, a rough allowance of time and requests, so slow pages or bloated sites get less attention. Content that only appears after client-side JavaScript runs is riskier: it can be missed or picked up late. A clean, fast, well-linked structure is what lets a crawler capture everything without spending its budget getting there.
Common mistakes
The most common failure is self-inflicted: an accidental Disallow in robots.txt or a stray noindex tag that shuts a crawler out of pages you actually want found. Broken internal links, long redirect chains, and content that only renders after a JavaScript click cause the same problem more quietly. A missing or stale sitemap slows discovery of new pages. Heavy load times eat into the crawl budget, so deeper pages in a large site may go unvisited for a long time. And publishing the same content under multiple URLs creates duplicate-content confusion that dilutes which version gets treated as canonical. None of these are exotic problems — a routine technical crawl audit catches nearly all of them before they cost visibility.
Relation to AI recommendations
ChatGPT, Perplexity, and Google run their own crawlers — GPTBot among them — that fetch pages the same mechanical way search engine crawlers always have, then feed what they collect into training or retrieval systems behind AI answers. Block those bots and you opt out of that pipeline entirely. But being crawlable is not the same as being cited: Ahrefs has found that only about 6–8% of URLs ChatGPT cites also rank in Google's top 10 for the same query, and roughly 80% of ChatGPT's cited URLs don't appear in Google's top 100 results at all — AI citation runs on a noticeably different selection process than classic ranking. What correlates more strongly with getting cited is how often a brand is mentioned across the web (Ahrefs measured roughly three times the correlation of backlinks). A crawlable page is the prerequisite; it isn't the whole game.
Example
Picture a small tax advisory firm in Leipzig that publishes a thorough guide to a home-office tax allowance. The writing is solid, but the page shows up nowhere for weeks. The cause turns out to be a leftover Disallow rule from a site rebuild, quietly blocking the entire guides folder in robots.txt. No crawler could enter, so nothing there could be indexed. Once the rule is removed and a current sitemap is submitted, Googlebot revisits within days, the guide starts appearing in search results, and it later turns up as a cited source in an AI assistant's answer.
Common questions
What's the difference between crawling and indexing?
Crawling is the visit itself: a bot fetches a page and reads what's there. Indexing happens after, when that content is stored and made retrievable for search queries. A page can be crawled successfully and still never get indexed — a noindex tag is the usual reason.
Do I need an llms.txt file to get crawled by AI bots?
No. Google has stated plainly that no AI-specific file or markup is required for AI Overviews or AI Mode, and Google's John Mueller has confirmed no Google Search system reads llms.txt at all. An Ahrefs analysis of roughly 137,000 sites that published one found about 97% saw no measurable referral traffic from it. robots.txt and a working sitemap remain what actually controls crawler access.