Free tool
Can AI assistants read your site?
Paste a domain. We read its robots.txt the way AI crawlers do and show which assistants are let in and which are kept out.
Which AI crawler is which
Nineteen tokens, each one a line you might find in a robots.txt. The names are the ones the operators publish and the check matches them case-insensitively, the way the crawlers read them. Two of them are commonly confused: Googlebot fetches for Google Search, including AI Overviews, while Google-Extended governs Gemini training and grounding only — blocking the second does not affect the first.
Answer-engine crawlers: blocking one is a visibility decision
The crawlers behind what a customer sees when they ask. Blocking one of these is a visibility decision.
| Token in robots.txt | Operator | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results and their citations. |
| ChatGPT-User | OpenAI | Fetches a page when a ChatGPT user asks about it. |
| Claude-SearchBot | Anthropic | Indexes pages for Claude's web search. |
| Claude-User | Anthropic | Fetches a page when a Claude user asks about it. |
| PerplexityBot | Perplexity | Indexes pages for Perplexity's answers. |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user asks about it. |
| Googlebot | Google Search, including AI Overviews and AI Mode. | |
| Bingbot | Microsoft | Bing search and Copilot answers. |
| Applebot | Apple | Siri, Spotlight and Safari suggestions. |
| DuckAssistBot | DuckDuckGo | DuckDuckGo's AI-assisted answers. |
| MistralAI-User | Mistral | Fetches a page when a Le Chat user asks about it. |
| Amazonbot | Amazon | Alexa's answers. |
Training crawlers: blocking one is a licensing decision
The crawlers that build training corpora. Blocking one of these is a licensing decision, and it does not affect whether you are named in answers today.
| Token in robots.txt | Operator | What it does |
|---|---|---|
| GPTBot | OpenAI | Training data for OpenAI models. |
| ClaudeBot | Anthropic | Training data for Anthropic models. |
| Google-Extended | Gemini training and grounding. Blocking it does not affect Google Search. | |
| Applebot-Extended | Apple | Training data for Apple's models. |
| meta-externalagent | Meta | Training data for Meta AI. |
| CCBot | Common Crawl | The open web archive many models are trained on. |
| Bytespider | ByteDance | Training data for ByteDance models. |
Sources: the matching rules are those of RFC 9309, the Robots Exclusion Protocol, and the managed block this check flags is documented in Cloudflare’s managed robots.txt reference.
robots.txt is a request, not a lock. This shows what your file asks of each crawler; it cannot measure whether a crawler obeys, and a fetch a person triggers inside an assistant may not consult the file at all.
Questions about the result
What does this check read?
Two files on the domain you enter: robots.txt, which tells crawlers what they may fetch, and llms.txt, a plain-text summary some sites publish for AI assistants. It applies the robots.txt rules to the homepage path, following the same longest-match logic the major crawlers use. It does not crawl your pages.
Does blocking a crawler stop an AI assistant from mentioning my business?
Not by itself. An assistant can still name you from other sources: directories, reviews, news, other people's pages. Blocking the answer-engine crawlers stops the assistant from reading your own pages, so what it says about you comes from everyone but you.
Why does my robots.txt block crawlers I never heard of?
Most often because your CDN or host wrote those lines. Cloudflare's managed robots.txt, for example, prepends a block that disallows several AI crawlers and is enabled by default for many new zones. The check flags that block when it sees Cloudflare's markers, so you know where the rules came from.
Is robots.txt enforced?
No. It is a published request. Well-run crawlers honour it; a crawler that ignores it is not measured here. Fetches a person triggers inside an assistant, such as asking ChatGPT or Perplexity about a specific page, may not consult the file at all.
What should a business that wants to be recommended allow?
The answer-engine crawlers, at minimum: the search and user-fetch crawlers of OpenAI, Anthropic, Perplexity, Google and Microsoft. Whether to allow the training crawlers is a separate decision about licensing your content, and it has no effect on being named in answers today.
Is Google-Extended the same as Googlebot?
No, and confusing them is the most common mistake we see. Googlebot fetches for Google Search, which includes AI Overviews and AI Mode; disallowing it removes you from Google. Google-Extended governs Gemini training and grounding only, and disallowing it has no effect on Google Search or on your rankings. They are separate tokens and must be set separately.
How do I allow AI crawlers in robots.txt?
Add a group per crawler at the top of the file: a User-agent line naming the token, then Allow: / under it. Order does not decide precedence — a crawler obeys the group that names it and ignores the * group entirely, so a token you name is exempt from a site-wide Disallow. If your host or CDN writes the file, edit it there rather than in your site's code, because a managed file is regenerated and your change is overwritten.
The file says whether assistants may read you. The visibility check measures whether one of them, Claude with live web search, names you.
See if AI names you