robots.txt
robots.txt is a plain-text file at the root of your domain that tells automated crawlers which parts of the site they may fetch and which they may not. It works like a sign at the door, not a lock on it: well-behaved crawlers, including the major AI bots, read the file and follow it voluntarily. It governs crawling access. It says nothing about whether a page gets indexed or cited.
Why robots.txt matters for AI visibility
Every large AI assistant now runs its own crawler, and each one checks robots.txt before it fetches anything. If a rule blocks that crawler, your pages simply cannot be read for that system, no matter how good the content is. This matters more than it used to because AI answer surfaces now carry real reach: ChatGPT reported 900 million weekly active users in February 2026, Google's Gemini app passed 1 billion monthly active users in August 2026, and Google AI Overviews reaches over 2 billion monthly users across more than 200 countries. A robots.txt written years ago for SEO purposes, before any of these crawlers existed, can quietly shut a brand out of all of them. Reviewing and deliberately opening the file to the crawlers you want is now a basic step, not an edge case.
How robots.txt works technically
The file always lives at your-domain.com/robots.txt and is made of simple rule blocks. Each block starts with a User-agent line naming a specific crawler, followed by Disallow lines (paths that crawler may not fetch) or Allow lines (explicit exceptions). A wildcard User-agent of * applies to any crawler that has no block of its own. The file is a request, not an enforcement mechanism: crawlers that respect the standard, including GPTBot and most other named bots, honor it; crawlers that ignore it face no technical barrier from robots.txt alone. You can also point to your XML sitemap here so crawlers discover your pages faster. Crucially, blocking a URL in robots.txt stops crawling, not indexing — a disallowed page that's linked from elsewhere can still show up in search results with no snippet. To actually keep a page out of results, use a noindex meta tag or header instead.
Common mistakes
The costliest mistake is a blanket Disallow: / left over from staging that quietly ships to production and blocks the whole site. A close second, now that AI traffic matters, is disallowing GPTBot or another AI crawler by name without realizing it, which removes any chance of being cited by that assistant. Some teams also treat robots.txt as a way to hide sensitive data — it does the opposite, since the file is public and effectively lists the paths you didn't want found. Typos in paths, or placing the file anywhere other than the domain root, silently make the whole thing inert. And a newer mistake is over-investing here: Google's own guidance states plainly that no special file, markup, or schema is required for AI Overviews or AI Mode, and warns against building separate "for AI" infrastructure instead of just keeping the real site crawlable. Check the file with Google Search Console's tester after any change, and keep it under version control.
Relation to llms.txt and GEO
In Generative Engine Optimization, robots.txt is the first gate: nothing an AI crawler can't read can end up in a citation. This is where robots.txt and llms.txt get confused. llms.txt is a proposed file that summarizes a site's key pages for language models to consume; robots.txt controls crawler access and is honored, at least in part, by real crawlers today. The evidence for llms.txt is weak — Google's John Mueller confirmed in 2025 that no Google Search system reads it, and an Ahrefs analysis of roughly 137,000 sites that published one found about 97% saw zero measurable referral traffic tied to it. robots.txt, by contrast, has an immediate, mechanical effect: block a crawler here and it cannot fetch your content at all. If you're deciding where to spend effort, get robots.txt right first — allow the AI crawlers you want, keep the sitemap reference current — before treating llms.txt as anything more than optional and unproven.
Example
A mid-size outdoor-gear retailer in Portland notices it never comes up as a source when customers ask ChatGPT or Perplexity for buying advice, despite having thorough product guides. Checking robots.txt turns up a line from an old site migration: User-agent: * with Disallow: /, never removed after the test environment went live. It had been blocking every crawler, AI and search alike, for over a year. The team removes the line, adds an explicit Allow for GPTBot and PerplexityBot, and re-adds the sitemap reference. Within a few weeks, both engines are able to crawl the guides again and the retailer starts showing up in relevant AI answers.
Common questions
Can I block one AI crawler without affecting the others?
Yes. Add a dedicated block for that crawler's user-agent, for example User-agent: GPTBot followed by Disallow: /, and it applies only to that bot. Every other crawler, including other AI assistants, follows whatever rules apply to it or to the wildcard block. Just be clear that doing this removes your visibility in that specific assistant's answers.
If I disallow a page in robots.txt, is it guaranteed to stay out of Google?
No. Disallow only stops crawling. A page blocked in robots.txt but linked from elsewhere can still appear in Google's index, typically without a preview snippet, because Google can see the URL exists even without fetching it. If you need a page reliably excluded, use a noindex meta tag or HTTP header, which requires the page to be crawlable so the instruction can actually be read.