Three places an advisor's facts live, and which ones AI crawlers reach

Three places an advisor's facts are published

The same firm is described in three places, and each has its own publisher and its own door. An AI crawler asking for a page in each gets three different results, set out below.

A fetch made for an answer can reach the first place and the third. It cannot reach the second, and that is what the second is for. Directory profiles and the Business Profile make a fourth kind of place, pages the firm fills in on someone else's site, and directories and review portals covers those.

The six tokens OpenAI and Perplexity document

OpenAI's crawler documentation names four user agents and Perplexity's names two. Each has one job, and a robots.txt group addressed to one of them does not bind the others, as RFC 9309 sets out. The descriptions in the table are the operators' own, shortened.

Anthropic runs three of its own, and its help center says all three honor robots.txt: ClaudeBot for training, Claude-User for fetches when a Claude user asks a question, and Claude-SearchBot to improve its search results.

Two of the six act for a person rather than as crawlers: ChatGPT-User and Perplexity-User, and their operators say robots.txt may not bind them. For an advisor, that answers the question of what guards client files before anyone asks it: nothing private can depend on a line in robots.txt.

Why the portal and the vault stay private without a robots.txt line

A crawler that requests an address inside the portal receives what a stranger receives: the sign-in page. The plans, statements, questionnaires and uploaded documents behind it need an account, and no crawler holds one. The login is the protection, and it holds whatever robots.txt says.

A robots.txt line adds nothing to that, and it can take something away. The file is public, so a list of the paths where client documents live is as easy for a stranger to read as for a crawler. Leave private paths out of it. And if the portal runs on a provider's host, your file does not speak for that host at all, because each host's robots.txt covers that host and no other.

What does expose client material is a copy made outside the login. A statement attached to a public page for one client, a plan exported to a shared link that opens without signing in, a folder on the web server that lists its contents: each is a public address that happens to be hard to guess. Hard to guess is not private. Client documents belong in the portal, and a copy anywhere else is a problem to fix that day.

The regulator's register: a page about you that you do not write

The register states the firm's registration in the regulator's words, on the regulator's site. A client who wants to check an advisor can open it, and so can a crawler, wherever the regulator lets one in. That makes it a second published account of who the firm is, beside the firm's own site, and the only one of the two the firm cannot edit.

Keep the firm's name on the site identical to the name on the register. A trading name on the site and a legal name on the register, with nothing on either page to connect them, leaves a reader with two names and no evidence that they belong to one entity. Put the registered name beside the trading name in the footer and on the about page, give the registration number with it, and link the entry.

The same goes for the address and the people. If the register lists an office the firm has left, or an advisor who has moved on, the site and the register now describe two different firms. Find out which one is out of date. If it is the site, change the site. If it is the register, ask the regulator how its record is corrected; each runs its own process, and this page describes none of them.

In the markup, the register entry can be the url of the Certification that records the registration, as the overview for financial advisors sets out. The visible link does more work: it is the one a client clicks to check the fact.

Nothing private on the public pages

Every public page is open to the crawlers above, and an assistant with web search on can quote what it fetched in a reply to anyone. So read each public page with one question in mind: could a client be recognized from it, by themselves or by someone who knows them? Client details can reach an advisory site in several ways.

• A case study that gives a client's age, profession, town and the size of their accounts, even with the name left out.
• A screenshot of a plan, a statement or a dashboard, whether or not the figures are blurred.
• A list or a logo wall of the businesses whose owners are clients.
• A newsletter archive that quotes a client's question in full, circumstances included.
• A file uploaded to the website's media library to send to one client, at an address anyone can open.

Describe the work instead of the person: what the firm does for owners preparing a sale, not what it did for one owner. Which details count as personal data, and what the firm owes its clients about them, depends on where it works; the compliance officer, or whoever runs the firm's privacy procedures, decides, and anything already public that should not be comes down first. How to state the fees, which do belong in public, is in stating how an advisory firm is paid.

Checking all three places from outside the office

In our audit of 9,483 small business websites (40 trades, 90 metros in the US, Canada, Europe and the UAE, late August to early September 2026), 3.2% disallowed AI crawlers in robots.txt on purpose, 7.2% served our crawler a challenge page instead of the site, and 3.0% rendered their content only in JavaScript. Each figure covers all 40 trades; we publish no advisor-only cut. The checks below find out which group, if any, your own site is in, and then look at the other two places.

1. Open robots.txt on your domain and find any groups naming OAI-SearchBot, GPTBot or PerplexityBot. A token with no group of its own follows the * group, so a * group that disallows everything asks OAI-SearchBot and PerplexityBot to stay out along with the rest. The crawler access check reads the file for nineteen tokens and reports which may fetch the homepage.
2. From a connection outside the office, request the fees page as a crawler would and read the status line first: curl -sI -A "PerplexityBot" https://www.example.com/fees. A 403 or 503 is a refusal. A 200 can still be a challenge page, so fetch the body as well and make sure it is your page. The status code tells you which case you are in.
3. Sign out of everything, open a private window and go to the portal's address. The sign-in page, and nothing else, is the right result.
4. Search your own site for files that belong in the portal: the media library, the sitemap, any folder that lists its contents. Anything carrying a client's name or figures comes down.
5. Open your register entry beside your site's footer and compare the name, the registration type, the number and the address, character for character.
6. With search on, ask ChatGPT and Perplexity how your firm is registered and how it is paid, then open the sources each answer links. They show which pages the answer cited, and whether any of them are yours.

These checks show whether the pages can be read. Whether an answer names the firm is a separate reading, taken with a fixed panel of the same questions each time, as our methodology describes.