Perplexity

Perplexity is an AI answer engine that runs a web search for each question, reads the pages it finds and writes a cited answer from them. It fetches at question time, so what a page serves in that moment is what it can quote.

Two crawlers with different jobs

Perplexity’s crawler documentation names two user agents. PerplexityBot crawls to build the index its search results draw on, and it follows robots.txt, so a disallow rule keeps a site out of those results. Perplexity-User fetches a page when a person has asked a question that needs it, and the documentation says it generally does not follow robots.txt, on the grounds that a person asked. Both publish their IP ranges, so a site can also tell them apart at the firewall. The split matches OpenAI’s between OAI-SearchBot and ChatGPT-User: one token for the index, one for the fetch a user triggered.

What that means for a page

A robots rule aimed at PerplexityBot removes a site from Perplexity’s index without stopping a user from having its pages read. A bot-protection wall stops both, since neither completes a challenge. In this site’s own audit of 9,483 small-business sites, 3.2% disallowed AI crawlers on purpose and 7.2% sat behind bot protection that served a challenge page instead of the site; the second group is blocking answer engines without having decided to. Past the fetch, the conditions are the same as for any generative engine: a passage that answers the question on its own, and one name, address and phone number stated the same way everywhere. See the 9,483-site audit.

What has been measured here

This site’s published probe of 9,240 businesses used Claude with web retrieval. It has not published a Perplexity rate, and the Claude figure is not one. What the free crawler check does test is the part that is engine-neutral: whether the pages a fetcher asks for come back as HTML, and what the robots file says to each named token.

A common question

What is the difference between PerplexityBot and Perplexity-User for a site?

One builds the index, the other fetches on demand. PerplexityBot crawls to assemble the index the search results draw on and it follows robots.txt, so a disallow rule keeps a site out of those results while leaving a person free to ask a question that causes its pages to be read. Perplexity-User handles that fetch, and the documentation says it generally does not follow robots.txt, on the grounds that a person asked for it. A bot-protection wall stops both, since neither one completes a challenge.

Sources

Crawler names and behavior: Perplexity, crawlers and OpenAI, crawlers. The audit figures are this site’s own; sample drawn per metro and trade, percentages describe the sample.