Tax firms and ChatGPT: open the public pages, keep the portal shut

OpenAI's crawler tokens, and the one to check first

OpenAI's crawler documentation names four user agents, each with its own job. One, OAI-AdsBot, only visits pages submitted as ads, which leaves three that matter to a tax firm. OAI-SearchBot indexes pages for ChatGPT search, and OpenAI states that a site disallowing it is not shown in ChatGPT search answers, though it can still appear there as a navigational link. ChatGPT-User fetches a page when a ChatGPT user asks about it: the client who pastes your fees page into a chat and asks what a return costs. GPTBot collects training data, and disallowing it, OpenAI says, does not remove a site from search answers.

The file where a site states its rules for all three is robots.txt, and it is read one token at a time. Under RFC 9309, each group of rules belongs to the tokens named at its head and to no others. A crawler that finds no group naming its own token follows the group for *, so in a file that names none of the three, OAI-SearchBot and GPTBot both obey the * group, and a * group that disallows everything shuts ChatGPT search out with the rest.

The file a tax firm needs is short. The search token is allowed on the public site. The portal's path is disallowed for everyone, which saves crawlers a wasted trip to a login screen and protects nothing, since the login does the protecting. The training decision sits in a group of its own, written on purpose, whichever way the firm decides it.

# Public pages open to ChatGPT search; the portal's login does the protecting
User-agent: OAI-SearchBot
Allow: /
Disallow: /portal/

# Training data: the firm's own decision (this example opts out)
User-agent: GPTBot
Disallow: /

# Everyone else
User-agent: *
Disallow: /portal/

Sitemap: https://www.example.com/sitemap.xml

Put the path your portal uses in place of /portal/, or drop those lines if the portal runs on another host, since a robots.txt file speaks only for the host that serves it. And the file is public, readable by anyone. A list of confidential paths in it is a map, so leave them out.

The portal is closed by its login, not by robots.txt

Behind the login sit the things no crawler should ever read: uploaded documents, signed engagement letters, prior returns, messages about a client's affairs. A crawler that requests a portal address receives what any stranger receives, which is the login screen. That is the protection, and it holds whatever the robots file says. Nothing about the portal needs to change for ChatGPT.

The weak points are copies that live outside it. A shared-file link that opens without signing in is a public page with a long address; if it is ever linked from a public page, or pasted anywhere a crawler can follow it, it can be fetched like any other page. The same goes for a file dropped into the website's media library for convenience, or a server folder that lists its contents. Client documents go in the portal and nowhere else.

The public side still owes a prospective client an account of the portal, because it is part of how the firm works: what the portal is for, how a new client gets an account, what to do when locked out, and the number to call. That page carries no client information, and a fetcher can read every word of it.

Files and widgets that sit between the two

Between the public pages and the portal is a third group: files and embedded tools that look public but may hand a fetcher nothing to read. A tax firm's site can hold several of them, and the table takes them one at a time.

Four of the ways a site keeps its words from a fetcher were counted in our audit of 9,483 small business websites (40 trades, 90 metros, late August to early September 2026), and each figure describes the whole sample rather than tax firms:

• 7.2% served our crawler a challenge page instead of the site
• 3.2% disallowed AI crawlers in robots.txt on purpose
• 3.0% rendered their content only in JavaScript
• 2.1% kept real content, a price list or a service menu, in a PDF

The audit publishes one number for tax advisors alone, and it concerns markup rather than access: 31.0% of their sites carried no schema.org markup at all, against 17.8% across all 40 trades.

What a tax firm's public pages carry for ChatGPT search

The facts themselves are listed on the tax advisors overview. What follows is the form they need, so that a fetch of the page returns them.

• In the HTML the server sends. Open the page with scripts off; what remains is what a fetcher gets.
• One public address per service and per situation, each linked from the services page, so a crawler that starts at the homepage reaches it. Internal linking is the route it takes.
• The fee basis in sentences, beside the service it prices, with the figures the firm holds steady and a note of what moves the rest.
• Each person's designations spelled out, with the issuing body and a link to its public register.
• The season in dated text: taking new clients or not, the firm's own intake date, and what happens after it.
• The firm's name, address and main number in one form on every page.
• AccountingService markup in JSON-LD that repeats the page and states nothing the page does not.

Nothing on that list was written for ChatGPT in particular; it is what any fetcher can use, whoever operates it. Nor is there any form, file or listing we know of for submitting a firm to ChatGPT's answers. The public pages are the submission.

Checking access from outside the office, in order

Three questions, in this order: may OpenAI's crawlers fetch the public pages, can they reach them, and do the pages hold the facts once they get there. The first step answers the first question, the next two the second, and the last three the third.

1. Permission. Open the robots.txt file at the root of your domain and find the groups, if any, headed by OAI-SearchBot, ChatGPT-User or GPTBot. If the file names none of them, OAI-SearchBot and GPTBot take their rules from the * group instead. The crawler access check does the same reading for nineteen answer and training tokens at once.
2. Reachability. Work from a connection outside the office, since a firewall may treat the office network as trusted. Request the services page with the search crawler's user agent and save the response: curl -sL -A "OAI-SearchBot" -o services.html -w "%{http_code}" https://www.example.com/services. Read the HTTP status code before anything else. A refusal shows up as a 403, 429 or 503. A 200 proves less than it seems, since a challenge page can arrive with one, so the saved file has to be your page and not a puzzle.
3. Reachability as a stranger. Sign out of the portal, open a private window and click through the public pages the way a stranger would. A public page that bounces you to the login screen, or to a challenge, does the same to every crawler.
4. Content. Turn scripts off in the browser and reload the services, fees and contact pages; whatever disappears was never in the HTML a fetcher gets.
5. Give ChatGPT the address of your fees page and ask what the firm charges for a [return type]. Opening a pasted address is ChatGPT-User's job, so a reply that cannot read the page, or misstates the fee basis, sends you back to the page itself.
6. Put three client situation questions to ChatGPT with search on, and follow the links under each answer to its sources. Which pages appear there, yours or other firms', is what the monthly panel then tracks, as being named for the client's situation sets out.

On the second step: a bot service may judge a request by the network it comes from as well as by the name it gives, so a test from your laptop and a fetch by the real OAI-SearchBot can be treated differently. Where the service keeps a log of OAI-SearchBot requests, that log settles it.

Access is a precondition, not a result. The checks tell you that ChatGPT can reach and read the firm; only the panel, month after month, tells you whether it names the firm for the questions clients ask.