Hallucinated features: find and correct wrong claims about your product

What a wrong claim about software looks like

Wrong claims about a software product come in a few shapes, and each shape points at a different kind of page. The general method, for any business, is in when AI gets facts about your business wrong. This is the software case, where the facts are features, plans and limits that change with every release, and the SaaS and software hub covers what the product's own site has to state in the first place.

The first row can have no source at all. Nothing you published says the product has the feature, and nothing says it lacks it, so an answer that states the feature anyway is what the glossary calls a hallucination. An answer to "does [Product] work offline" can repeat "no" only where a page says no. That makes an invented feature a documentation problem, and the documentation is yours.

Ask a fixed set of questions about your own product, every month

Write ten or so questions about your own product in the words a buyer would type, and put the same ones to ChatGPT, Claude, Gemini and Perplexity every month. The set covers every kind of fact that a release can change:

• Features: "does [Product] have [a feature buyers in your category expect]", including features you do not have, since that is where an invented yes would show.
• Integrations: "does [Product] integrate with [the CRM your buyers use]", including one you have retired.
• Plans: "which [Product] plan includes [feature]", "is there a free plan of [Product]".
• Limits: "how many users can [Product]'s [plan] have", "what is the file size limit in [Product]".
• Security: "where does [Product] store customer data", "does [Product] have [the report buyers in your market ask for]".
• Price: "how much does [Product] cost for [n] users".
• Removals: "can [Product] still [do the thing you removed]".

Use a fresh conversation for each question and keep the wording fixed, so one month compares with the next. Run each question more than once, because answers vary between runs; the way we measure asks each question once per dated reading, of Claude alone, and with monitoring reads the pattern across months. The panel's file and record format are in setting up prompt monitoring. What differs here is the scoring. A question about your own product names you already, so it is not asking whether you are named; it asks whether the facts in the answer are right.

Log each wrong claim with the answer's own words and its source

Each wrong claim gets a row: the date, the assistant, the question, the claim copied exactly as the answer worded it, and the source the answer showed, if it showed one. A spreadsheet is enough. The columns matter more than the tool:

date,assistant,question,claim_as_worded,source_shown,claim_type,our_page_with_the_fact,fix,fixed_on,next_month
[YYYY-MM-DD],[assistant],"does [Product] integrate with [tool]","[the sentence, copied exactly]",[URL or none],retired integration,[URL or none],[what you changed],[YYYY-MM-DD],[still wrong / right / not asked]

Copy the wording exactly. "Integrates with", "supports" and "can connect through a connector" are three different claims, and the exact phrase is what you will search for when you trace it. Where the answer cites a page, record the address even if the page looks harmless; a source citation leads straight to the page to check.

Read the column for your own page first. If it says "none", no page of yours states the correct fact, and that is the first fix, because nothing can correct the claim until a page states it. Then rank the rows. A wrong price or a wrong security statement costs more than a wrong detail about the interface, and each row gets an owner: whoever owns the page that has to change.

Trace each claim to the page it came from

When the answer shows a source, open it and find the sentence. When it shows none, search for the claim's exact wording, starting with your own site, because the pages you wrote and forgot are the ones you can fix today. In order:

1. Open the source the answer cited, find the sentence, and note whether the page is yours.
2. With no source shown, search your own site, docs, help center and blog for the exact phrase, then for the feature's name next to an old plan name.
3. Fetch the old addresses directly: old pricing URLs, retired campaign pages, the integrations directory, old versions of the docs. A page you thought was gone may still return one.
4. Then read the pages others wrote: review-platform listings and their feature grids, integration marketplaces, partner pages, forum threads, comparison pages by other companies.
5. If nothing anywhere states the claim, treat it as invented and go straight to writing the page that states the fact.

For retired pages, Google's documentation describes both routes. Its page on redirects gives a removed page whose visitors you want to send somewhere new as a reason to redirect, and says a permanent redirect shows the new target in search results. Its page on HTTP status codes says a URL that now returns a 4xx code is dropped from the index over time. The glossary entries on the 301 redirect and the HTTP status code cover the difference. Either way, your own site stops serving the old sentence.

Publish what the product does not do, and date every removal

The page that corrects an invented feature states the negative: what the product does not do, on which plans, and what to use instead. It can be a section on each feature's page or a page of its own, and it is written as plainly as the features are:

• "[Product] does not work offline. The mobile app needs a connection to save changes."
• "There is no self-hosted edition; [Product] runs only in our cloud, in [regions]."
• "Single sign-on is on the [plan] plan and above, not on [lower plan]."
• "[Product] does not sync [object] with [tool]. You can export a CSV and import it there."

Each removal gets a dated entry in the changelog ("Removed: [feature], [date], all plans; use [replacement]"). The feature's old docs page keeps its address with a dated notice saying it was removed and what replaced it, or redirects to the page that says so. The changelog, docs and trust pages that carry these dates have an article of their own, and the rows of your comparison pages change on the same day.

Stating a gap can feel like handing a competitor a sales line. It is also the only way an answer to "does [Product] do [x]" can say no with your page behind it.

What changes after a fix, and what nobody can change

Fixing the page changes the source, not the answer. An assistant can repeat an old fact after the page has changed, and no one, the vendor included, can edit an answer directly.

It can repeat the old fact for more than one reason. A model's training data was collected before the change, up to its knowledge cutoff, and an answer written from that data alone carries what was true when it was collected. A search index holds its copy of a page until it fetches the page again. And other pages may still say the old thing. Anthropic's documentation for Claude's web search tool says it gives Claude current web content beyond its knowledge cutoff, with citations to the sources, so an answer written with a fresh search can carry the change.

No operator publishes how long a changed page takes to reach its answers, as far as we have found. OpenAI's crawler page gives one timing, about 24 hours, and it is for a robots.txt change to take effect in search results, not for a change to a page's content.

So the check is next month's run of the same questions. A claim that is gone after the fix suggests you found its source. A claim that stays points to a source you have not found yet, and one that goes on one assistant but stays on another points at what that assistant can still fetch. Anyone who offers to edit what an assistant says about your product is offering something nobody outside the operator can do.