robots.txt for AI Crawlers: Controlling GPTBot, ClaudeBot, and the Rest

What robots.txt Actually Does

The robots.txt file sits at the root of your domain — at yourdomain.com/robots.txt. Any bot visiting your site is supposed to fetch this file first and read which areas it's allowed into. The convention dates back to the 1990s, when it governed classic search engines. What's changed is that today, AI crawlers such as GPTBot or ClaudeBot check the same file before crawling.

It helps to be clear-eyed about what this actually is: a request, not a lock. Reputable bots follow the instructions because their operators have committed to doing so — nothing stops a bot technically from ignoring the file. The major AI providers do publish their crawlers' names openly and generally honor the rules, because getting caught disregarding them would be a reputation problem they don't want.

The format is deliberately simple. User-agent names a specific bot, and the Disallow or Allow lines beneath it set the rule. An asterisk as the user agent addresses every bot at once. That simplicity is the whole point — a few lines can steer the behavior of dozens of crawlers without writing any code.

Why AI Crawlers Need Their Own Rules

Classic search bots crawl your page to index it and rank it, and in exchange they send you visitors. AI crawlers work differently: they gather content to train language models, or to answer a question directly inside a chat interface. The user gets the answer without ever clicking through to your page. The old trade — visibility for traffic — doesn't hold the same way anymore. Pew Research found that when an AI summary appears above a Google result, users click through to a traditional result only about 8% of the time, versus 15% when no summary shows.

That's why AI crawlers raise different questions than Google ever did. An online shop might want its product pages surfaced in AI answers but doesn't want years of guide content quietly absorbed into someone else's training set. A law firm may not want any client-related content processed by a model at all. A publisher selling licenses wants to stop free scraping outright. The right answer depends entirely on what you're protecting and what you're trying to be found for.

A second wrinkle: some providers split training from live answers into separate bots entirely. OpenAIruns GPTBot for training and a distinct user agent for real-time answers inside the ChatGPT product. That split lets you make a finer call — block training while still allowing your content to surface in live, cited answers.

The AI Crawlers You Need to Know by Name

For any rule to actually work, you need the exact user agent name. Get it wrong and the rule does nothing — the bot just keeps crawling unhindered. Names change and new bots appear, so it's worth checking each provider's official documentation periodically rather than trusting a list copied from a forum — those go stale fast and often carry wrong names.

Here are the crawlers worth knowing right now. Keep in mind a single provider can run several bots at once, depending on whether it's for training, indexing or answering live queries.

If you're not sure which bots are actually hitting your site, check your server logs. You'll see the real user agent strings of every access, so you can tune robots.txt to what's actually visiting instead of guessing at a list where half the names never show up.

• GPTBot — OpenAI's crawler, gathers content for model training
• OAI-SearchBot — OpenAI's bot for search and answer features in ChatGPT
• ClaudeBot — Anthropic's crawler for training the Claude models
• Claude-Web / Claude-User — Anthropic's fetches made in response to user queries
• Google-Extended — controls use by Google's AI products, separate from standard Googlebot
• PerplexityBot — the crawler behind the answer engine Perplexity
• CCBot — Common Crawl's crawler; its dataset feeds training for many models
• Bytespider — ByteDance's crawler, often generating heavy request volume
• Meta-ExternalAgent — Meta's crawler for AI training purposes

Ready-to-Use Rules You Can Copy

The simplest case is blocking one bot outright. Put its name under User-agent and set Disallow to a single slash, which stands for your entire domain. For example: User-agent: GPTBot, then Disallow: /. That shuts GPTBot out of your whole site. Repeat the block for each additional bot you want to stop.

Often you'll want finer control. A training provider, for instance, might want its public course pages crawlable but its members' area and downloads off-limits. In that case, allow the bot by default and block only specific paths: Disallow: /members/ and Disallow: /downloads/. Everything else stays open. Combining targeted Disallow lines with optional Allow exceptions gives you precise control without blocking the whole site.

Watch the syntax details. List one path per line, separate each bot's block with a blank line, and the order of the blocks doesn't matter. A common mistake is stuffing several bot names into one User-agent line — that doesn't reliably work. Write one clean block per bot instead, even if the file ends up longer.

• Block every AI bot: a separate block per bot, each with Disallow: /
• Bar the entire site for everyone: User-agent: * plus Disallow: / — careful, this blocks Google too
• Protect a single folder: Disallow: /path/ under the relevant bot
• Carve out an exception: Allow: /path/public/ after a broader Disallow

What robots.txt Cannot Do

robots.txt doesn't protect anything — it signals intent. Bots that don't respect the rules simply ignore your entries, and lesser-known or aggressively run crawlers do this routinely. If content is genuinely confidential, put it behind a login or a server-side access block, not just a Disallow line in a public file. Disallow stops crawling; it does nothing to stop anyone who already has the URL from loading it directly.

A second misconception involves models already trained. Blocking GPTBot today only stops future crawls — content it already read before the block can't be pulled back out of a trained model. robots.txt works going forward, never retroactively. If you wait too long to act, a large share of your content may already be sitting in someone else's training data.

Third, blocking crawlers changes nothing about what other people publish about you. If your company is covered in press articles, directories, or forum threads, that external content still ends up in training data regardless of your own robots.txt. You only control access to your own domain, not everything the web already says about you. Ahrefs research even found that how often a brand is mentioned elsewhere on the web correlates more strongly with AI citation rates than backlinks do — third-party mentions matter more than most site owners assume.

Block or Allow: The Real Trade-off

The reflexive move for a lot of site owners is to block every AI bot. It's the easy choice, but rarely the right one. More people are asking questions directly inside AI assistants instead of running a classic search — ChatGPT alone reached 900 million weekly active users in early 2026, and Google's AI Overviews now reach over 2 billion users a month. If you've blocked everything, you simply never get named in those answers. For a local business or trade service, that can mean going invisible in a growing share of the queries that matter.

So it makes sense to separate training from answering. Many site owners block pure training bots like GPTBot or CCBot while allowing the answer and search bots that pull content live and typically cite it back to a source. That way you keep your content out of silent training pipelines while staying present in the AI answers that can actually bring you visibility and inquiries.

The right call depends on your business model. A publisher whose revenue comes from licensing content has good reason to block hard. A service business trying to win customers usually benefits more from being findable. Make the decision deliberately and write down why, rather than copying a standard robots.txt template built for a completely different situation.

Setting It Up, Testing It, and Keeping It Current

Save the file as plain text named robots.txt in your root directory so it's reachable at yourdomain.com/robots.txt. Loading it in a browser tells you right away whether it's being served correctly. In most content management systems, you can edit it through an SEO plugin or by editing the file directly. Make sure the URL doesn't redirect and doesn't return an error code — either one causes bots to ignore the rules entirely.

Check that every change actually takes effect. The easy way is to load the file and compare each block against what you intended, line by line. The real proof comes from your server logs, though — that's where you see which bots are actually hitting your site and whether a supposedly blocked crawler still shows up. If a blocked bot keeps appearing, either the name is wrong or the bot is ignoring the rule.

Treat robots.txt as a living document, not a set-and-forget file. Providers rename bots, new crawlers show up, and your own strategy shifts over time. Putting a recurring check on the calendar — quarterly works well — to review the official bot lists against your own logs keeps the file current. Otherwise a block you set up two years ago may be doing nothing today because the bot behind it now operates under a different name.