AI Engines · 8 min read · September 25, 2026 · Hafsa Tariq, gaash.ai
A fulfillment center's public pages and portal: what crawlers can reach
A fulfillment center's site has two halves: public pages anyone can fetch, and a client portal behind a login. Crawlers reach only the first, so the facts an e-commerce founder asks about (the platforms you connect to, storage and pick-and-pack pricing, cut-offs, returns handling, locations) belong on public pages as text, while order data, inventory and client dashboards stay behind the login.
Two sites under one domain: the public pages and the portal
The public half is the services, pricing, integrations, locations and returns pages. The portal is orders, inventory, SKU lists, billing and dashboards, at an address that answers with a login. An AI crawler arrives with no account, so it receives the public half and the login page, and nothing past it.
That split sets what a fetcher can read when a founder asks. With web search on, an assistant reads a handful of pages before it answers "which fulfillment center near Toronto connects to our store platform and ships the same day", and the pages it can read are public ones: yours, a directory's, another warehouse's. Whatever you want said about your integrations or your prices has to be in the public half.
What OpenAI's four user agents do, as OpenAI documents them
OpenAI's crawler documentation lists four user agents, each with its own job. Three of them touch a fulfillment center's public pages; the fourth visits only pages submitted as ads.
| User agent | What OpenAI says it does | What it means for your site |
|---|---|---|
| OAI-SearchBot | Indexes pages for ChatGPT search. A site that disallows it is not shown in ChatGPT search answers, though it can still appear as a navigational link; changes take about 24 hours | The token your public pages admit if you want them in ChatGPT search answers |
| ChatGPT-User | Fetches pages for actions a user starts; not used for automatic crawling or to decide search inclusion; robots.txt rules may not apply to it | It can open a page a user points it to, but without a client's credentials it receives what any stranger does |
| GPTBot | Collects training data; disallowing it does not remove a site from search answers | A decision about training, separate from being found |
| OAI-AdsBot | Visits only pages submitted as ads | Nothing to decide unless you submit pages as ads |
For a portal, the row to read closely is ChatGPT-User. Because robots.txt rules may not apply to it, a line in the file cannot be relied on to keep it out; a login can. The robots.txt article sets out the full token table and a file you can adapt.
The other crawlers that reach a public page
Anthropic runs three robots: ClaudeBot collects training data, Claude-User fetches pages when a Claude user asks, and Claude-SearchBot indexes for search, and Anthropic says all three honor robots.txt. Perplexity says PerplexityBot surfaces sites in its search results and is not used for training, while Perplexity-User generally ignores robots.txt because a user asked.
Google keeps Googlebot, which feeds Search and so AI Overviews and AI Mode, apart from Google-Extended, which controls use in Gemini training and grounding and has no effect on Search. The comparison of AI search engines sets out who fetches for each assistant. The fetchers that act for a user differ on the file: Anthropic says Claude-User honors robots.txt, while OpenAI and Perplexity say theirs may not. For a fulfillment center one point holds at every operator: none of these fetchers has a client's password.
The facts a founder asks about, as text on public pages
| The founder asks | What the page states | Where it goes |
|---|---|---|
| Do you connect to my store? | Each store platform and marketplace by name ([store platform], [marketplace]), what syncs (orders, inventory, tracking), which way and how often | An integrations page, one entry per platform |
| What does storage cost? | [price] per [pallet, bin or cubic foot] per month, the minimum, and what the rate includes | A pricing page in HTML |
| What does pick and pack cost? | [price] per order, [price] per additional item, packaging at [price] or included, and the date the prices apply from | The pricing page |
| What else will I pay? | Receiving at [price] per [unit], account minimums, onboarding, and what triggers each fee | The pricing page, beside the main rates |
| When do orders ship? | "Orders in by [time] [time zone] ship the same business day from [site]", with weekend and holiday cut-offs | Each location's page, and the pricing page |
| How are returns handled? | Inspection, restocking within [n] business days as your own measured figure, what happens to unsellable items under your policy, and the fee | A returns page |
| Where are you? | Each location's city, what ships from it, and its daily pickup times | A page per location |
| Can you handle my product? | Special handling you offer (kitting, lot and expiry tracking, temperature control, oversized items) and what you do not take | The service pages |
Write those facts into the HTML, not only into a quote calculator that a script builds or a rate card kept as a file. For an AI fetcher, usually no scripts run, and Google renders JavaScript only as a separate, deferred step. In our audit of 9,483 small business websites, 3.0% rendered their content only in JavaScript and 2.1% kept real content, such as a price list or a service menu, in a PDF (across all 40 trades; we publish no logistics-only cut); the article on price lists kept as files covers moving one onto the page. How to date the cut-off and the restocking time is in the article on operating figures.
What stays behind the login, and how to keep it there
Order data, inventory levels, SKU lists, client names, billing and dashboards belong behind authentication, and only there. A crawler has no account, so the login keeps it out; robots.txt does not. The file is public, it is a request rather than a lock, and listing the portal's paths in it tells anyone who reads it where they are.
Google's robots.txt introduction says a URL blocked in the file can still be indexed if other pages link to it, and its page on noindex adds that a page blocked in robots.txt cannot have its noindex read. So a page that is public but should stay out of search, such as an order-tracking page, gets a noindex and stays crawlable, and a page that must stay private gets a login. What a tracking page may show about a person is a question for your counsel, because privacy rules differ between the US, Canada, the EU and the UAE.
Then check the seams. The portal's address should serve a login and nothing else to a visitor without credentials, and no public page should link to a report, an export or a document stored inside it.
Checking what a crawler receives, in order
- Run the crawler access check. It reads your robots.txt for all 19 tokens and reports which are allowed and which are asked to stay out. It reads robots.txt only, so it cannot tell you what a firewall or bot protection does.
- Fetch a public page as a crawler would and read the status code:
curl -A "OAI-SearchBot" -I https://www.example.com/pricing. A 200 is the page. A 403, or an HTML challenge page, points to bot protection, which is set in your CDN or security plugin rather than in robots.txt; 7.2% of the sites in our audit served our crawler such a challenge. - Open the pricing and integrations pages with scripts turned off. Whatever disappears is what a fetcher that runs no scripts does not receive.
- Open the portal's address in a private browser window. It should show a login and nothing else.
- Add founder questions to your panel ("which fulfillment center near [city] connects to [store platform] and ships same day") and record the page each answer cites, as the article on reading answers by lane describes.
A founder chooses a fulfillment center from the shop's side of the table, and the overview for online shops shows what that side asks of its own pages. What the rest of a logistics site has to state is in the overview for logistics providers.
Common questions
Should we block GPTBot to protect client data?
Client data should never be reachable without a login, whatever robots.txt says, so blocking GPTBot protects nothing that matters there. GPTBot collects training data from public pages, and OpenAI's documentation says disallowing it does not remove a site from search answers. Whether your public pages go into training is a separate decision, written in its own robots.txt group.
Our client portal is on a subdomain. Does our robots.txt cover it?
No. Google's page on how it reads robots.txt says the rules apply only to the host, protocol and port where the file sits, so the file on your main domain says nothing about the portal's subdomain. The portal is protected by its login either way; a robots.txt of its own there would only be a request, and it would list the portal's paths for anyone who reads it.
Can ChatGPT read our client portal?
Not past the login. ChatGPT-User fetches a page when a user asks, and OpenAI says robots.txt rules may not apply to it, which is why the portal has to rely on authentication rather than on the file. A fetch that arrives without a client's credentials receives the login page, exactly what any stranger would see.
Should the integrations page list platforms we reach through a connector?
Yes, and say that it is a connector: the platform by name, that the link runs through [connector] rather than a native integration, what syncs (orders, inventory, tracking), how often, and who supports it when it fails. A founder asks whether you connect to their store, and a precise line answers that better than a row of logos with no text.
Sources
The percentages are this site's own, from the audit named below, and describe all 40 trades in the sample. What each crawler does is from its operator's own documentation, linked where it is used; the crawler access check reads robots.txt only.
Sources: our audit of 9,483 small-business sites, with its method · OpenAI, crawler documentation · Anthropic Help Center, Anthropic's web crawlers · Perplexity, crawler documentation · Google, common crawlers · Google, robots.txt introduction · Google, how Google interprets the robots.txt specification · Google, block indexing with noindex · Google, JavaScript SEO basics
Read on