What ChatGPT and Perplexity can reach on a recruiting agency's site

What OpenAI and Perplexity document about their six user agents

Both operators publish what each of their crawlers does, and everything below rests on those two documents. OpenAI's crawler page lists four user agents and Perplexity's lists two. The table gives each AI crawler as its operator describes it, shortened, with what a robots.txt rule does for it.

Neither page describes how an answer comes to name one agency over another, and nothing here depends on knowing it. The two pages settle access, and only access: which fetches a robots.txt line can turn away, and which ones it may not reach at all. Answers can also come from what a model learned in training, up to its knowledge cutoff; what follows is about the pages fetched when web search is on.

Three places an agency's pages live, and whose rules cover each

For robots.txt, the unit is the host. Google's description of the standard says a file's rules apply only to the host, protocol and port that serve it, and that a robots.txt on a subdomain is valid only for that subdomain. RFC 9309 adds that a group of rules addressed to one crawler does not bind another. An agency's pages can sit in three places, and the file means something different in each.

The middle row is the easiest to get wrong. A job host at a subdomain such as jobs.example.com looks like part of your site, but if the applicant-tracking system serves it, the file at jobs.example.com/robots.txt is whatever that system returns, and a rule in your main file does nothing there.

robots.txt is also only one gate. Bot protection is another, and no robots.txt reveals it: in our audit of 9,483 small business websites (40 trades, 90 metros, late August to early September 2026), 7.2% served our crawler a challenge page instead of the site. That figure covers the whole sample; the audit publishes none for staffing agencies or for job hosts.

Why each open role belongs on your own domain as text

Put every open role on a page of your own domain as well as on the job host: one page per role, in HTML, with its application link pointing to the host. That page answers to your robots.txt, keeps its address if you change systems, and can say in a sentence that your agency is the one recruiting. Each role page states:

• The job title as the job is known, with the contract type and the place: "[Title], [permanent, contract, temporary or temp to hire], [city]", or that the role is remote and from which countries.
• The hours or the shift pattern, and the pay where you publish it.
• What the work involves and what it requires, in the client's terms.
• The date posted, the recruiter who handles it, and how to apply.
• Your part in it: "[Agency name] is recruiting for this role on behalf of [the client, or a [industry] company in [city]]."

Whether a posting must show pay, and what it may say about requirements, depends on rules that differ by jurisdiction, and your counsel states them. A posting describes the work. It never describes who may apply in terms of a protected characteristic.

If you add structured data, Google's documentation for job posting markup sets the terms. It says to put JobPosting on the page for a single job and never on a list of jobs, on the same page as the description a job seeker can read; that the description must be complete; and that a posting must not need a login to view its details. Its required properties are datePosted, description, hiringOrganization, jobLocation and title. hiringOrganization is the company offering the job, and where a staffing agency is hiring for an employer that stays anonymous, the documentation gives confidential as the name.

Markup makes a page eligible for Google's job search experience and no more; Google's general structured data guidelines say a feature enabled by markup is not guaranteed to appear. Whether answers then credit a role to your agency or to the client is one of the record columns in AI visibility for recruiters.

A closed role comes down the day it closes

A filled role that still reads as open misleads every candidate who applies to it, and an answer built from last week's copy of that page can repeat it. Neither OpenAI's page nor Perplexity's says how long a fetched copy stays in an index, so the only safe closed role is one whose page no longer says it is open.

Google is explicit about job pages. Its job posting documentation says expired postings are not allowed, that a job no longer open for applications must be expired, that failing to act in time may result in a manual action, and that a job filled before its validThrough date should be removed. It names three ways to do it: set validThrough to a date in the past, remove the page so it answers with a 404 or 410, or take the JobPosting markup off the page.

The same day a role is filled or withdrawn:

1. Close it in the applicant-tracking system, so the job host stops taking applications.
2. On your own domain, remove the page so it returns a 404 or 410 status code, or replace the posting with a sentence saying the role is filled, with no JobPosting markup left on it.
3. Take it off your jobs list and your sitemap, and remove any link to it from other pages.
4. Close it on every board you posted it to, and write down the date you did.
5. Tell Google: its documentation recommends the Indexing API for job posting URLs, ahead of removing them from the sitemap.

What stays behind the login: CVs, candidate records and client briefs

The portal and the client area are closed by their login, because a crawler has no account to sign in with. robots.txt is no substitute. Every line in it can be read by anyone who opens the file, so Disallow: /cv-uploads/ announces the folder rather than hiding it, and the two fetchers that act for a person are the ones least bound by it: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.

Three things can slip past a login without anyone breaking it. A shortlist sent to a client as an open link opens for anyone who has the address, including a fetcher asked to summarize it by whoever pasted it into a chat. A CV stored at a public file address opens the same way. A client brief copied into a public posting, with the client's name, budget or reason for hiring, is public once the page goes up. Keep shortlists and CVs behind sign-in or links that expire, and write each posting from the brief instead of pasting it.

Google's job posting policies add a rule for the markup: a posting may collect CVs only for an open position, so a general page inviting candidates to send a CV is not a job posting and carries no JobPosting markup. How long CVs are kept, what consent a candidate gives and what a candidate may ask you to delete are set by data-protection rules that differ by jurisdiction; your counsel or regulator states them. The public site carries the agency's own policy in plain words, and the questions candidates ask about their CV are answered on the candidate pages.

Checking each host, from your domain to the portal

1. Run the crawler access check on your own domain. It reads the host's robots.txt and reports, token by token, whether the file lets each crawler fetch the front page; see whether OAI-SearchBot, GPTBot or PerplexityBot has a group of its own. Under RFC 9309, a crawler that finds no group naming it follows the * group.
2. Run it again with the job host's name. If the file there disallows OAI-SearchBot or PerplexityBot, the postings on that host are shut to those two crawlers, and your own copies of the roles are the ones they can fetch.
3. Open one role page on your own domain with scripts turned off, or view its source. The title, place, contract type and pay should be in the HTML; if they arrive only through a script from the job host, a fetcher that runs no scripts receives an empty frame.
4. Open the portal, a shortlist link and a CV link in a private window, signed out. Each should stop at a sign-in screen or refuse; any that opens is public.
5. Open the old address of a role you closed. It should answer with a 404 or 410, or say that the role is filled.
6. Ask Perplexity about one open role by its title and city, and read the sources it lists: your page, the job host or a board. That is the page a candidate who follows the answer lands on.

These checks show what the crawlers can reach, not whether an answer names the agency or to which side; the panel in AI visibility for recruiters records that. What the rest of the site should state for hiring managers and candidates is on the recruiting and staffing overview.