When your own bot protection turns the AI crawlers away: a check for MSPs

A firewall answers before robots.txt is ever read

Two separate layers stand between an assistant's crawler and your page. The first is permission: robots.txt, a file the crawler fetches and then obeys, token by token, the way RFC 9309 sets out. The second is access: the CDN, the web application firewall and any security plugin, which see the request first and can refuse it, challenge it or slow it down without consulting the file.

In our audit of 9,483 small business websites (40 trades, 90 metros, late August to early September 2026), 7.2% served our crawler a challenge page instead of the site, and 3.2% disallowed AI crawlers in robots.txt on purpose. Our crawler identified itself as gaash-audit/1.0 and was on no site's allow list, so the 7.2% describes what an unlisted crawler met. A site can challenge strangers and still let a crawler it recognizes through.

The layers can also collide at the robots file itself. RFC 9309 treats a robots.txt request that gets a 4xx answer as unavailable, which leaves the crawler free to fetch anything, and one that gets a 5xx as unreachable, which obliges the crawler to assume complete disallow. A protection rule that answers the crawler's request for /robots.txt with a 503 closes the whole site to a crawler that follows the standard, whatever the file says.

Which crawlers to let through, and what each operator publishes

An allow rule is for the answer-side crawlers: the ones that build an assistant's search index and the fetchers that open a page when a person asks. Training crawlers such as GPTBot and ClaudeBot are a separate decision, made in robots.txt, and the robots.txt article covers it. Each operator documents how to recognize its own.

Two rows matter more at the firewall than in the file. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, so for those two fetchers your protection is the only control in play. And Anthropic says its bots respect anti-circumvention technologies and will not attempt to bypass CAPTCHAs. A challenge stops Anthropic's crawlers where it stands, and lifting it for them is your decision.

Write the allow rule on the name and the address together

A user agent string is what the client says about itself. RFC 9110 defines the header as information about the software sending the request, and any client can send any string. Google's crawler page warns that the string can be spoofed, and Bing's calls it easy to spoof. A rule that admits a request because it says OAI-SearchBot admits anyone who types that name.

So match both. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges, and it publishes a separate list for each of its crawlers. Perplexity recommends the same for PerplexityBot. Anthropic publishes the source addresses its crawlers use, Google publishes ranges for its common crawlers, and Bing offers a tool to verify Bingbot. A rule that requires the token and an address from the operator's list is one a stranger cannot pass by renaming a script.

Then place and scope it. The rule has to be evaluated before the rules that challenge or block, in whatever order your tooling applies them, and it should cover the public hostnames and paths and nothing else. Check it against your rate limits too: a limit that answers with 429 tells Google's crawlers the server is overloaded, which Google's status code page says makes them slow down.

Keep the lists current. They are files the operators publish, and Perplexity's page says its addresses are updated regularly, so a rule built from a copy taken once goes stale without anyone touching it.

Test it: user agent, status code, page title, then the log

Test from a network your rules do not treat as trusted, with the user agent strings the operators publish. The loop below requests one page as four answer-side crawlers and prints the HTTP status code, the final address after redirects and the page title. The title is how you tell your page from a challenge that came back with a 200. OpenAI notes that the version number in its strings may change, and Anthropic's page gives no full string, so the loop sends that token alone.

URL=https://www.example.com/managed-it-services
for UA in \
  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
  "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  "Claude-SearchBot"
do
  curl -s -L -o page.html -w "%{http_code} %{url_effective}  " -A "$UA" "$URL"
  grep -o -i -m1 "<title>[^<]*" page.html; echo
done

Read the codes the way RFC 9110 defines them. A 403 means the server understood the request and refuses it. A 503 means the server is overloaded or down for maintenance, which it is not if a browser gets the page at the same moment. A 200 carrying someone else's title is a challenge page. Google documents what these do to its own crawler: content from a URL that returns a 4xx is not used, and an indexed URL that starts returning one is removed from Search, while a 5xx slows crawling and a URL that keeps failing is eventually dropped. None of the other operators' pages cited here says what a refusal does to its index.

Your test request is not the crawler. If the allow rule checks addresses, as it should, a request from your laptop carrying the OAI-SearchBot name is an impostor to it and gets challenged, which is the rule working. What settles the question is the event log: filter your protection's log to the operators' published address ranges and read the action taken on each request (allowed, challenged or blocked) and the path it asked for.

1. Run the crawler access check for the permission side. It reads your robots.txt, token by token, and nothing else, so it cannot see a challenge or a firewall rule.
2. Request /robots.txt and two or three public pages with each answer-side user agent, from outside your trusted networks, and record the status code, the final address and the title.
3. Read the event log for the operators' published ranges, path by path.
4. Change the rule if it needs it, test again, and date each test; repeat after any change to the protection's policy, a change of CDN plan or a site migration.

The client portal and remote-access tools stay shut by login and network rules

Nothing above loosens what protects clients. RFC 9309 says its rules "are not a form of access authorization", and its security section warns that listing a path in the file makes that path public, pointing to authentication for anything that has to stay closed. The client portal, the ticketing login and the remote-access and monitoring consoles stay shut by their logins and your network controls. They need no line in robots.txt, where a Disallow for the portal's path would tell every reader of the file where to look.

Scope matters where the two share a host. If the portal sits at a path on the marketing site, write the crawler allowance for the public paths only, so the login page keeps the rules it has now. If it runs on its own hostname, give that host its own policy and leave the crawler allowance off it. A crawler that does reach a login page receives the login page or a 401, which RFC 9110 defines as a request lacking valid credentials, and there is nothing on it to quote.

What belongs on the public side is the description: what the portal is for, how a new client gets access, the service levels, the systems you support. A buyer asks about those before signing anything. The IT service providers overview lists the facts a provider's pages carry, and the article on credentials covers the certificates and partner designations that go beside them.