How to measure AI visibility, and know when an assistant changed its answer

What exactly is being measured?

Not traffic and not rank. The thing a business wants to know is whether, at the moment a buyer asks an assistant for a recommendation, its name comes back, and in what company. That moment has four parts a person can record: was the business mentioned at all; was it recommended or only listed; was it first; and which other businesses appeared beside it. A fifth is what the assistant said about it, which is where wrong facts and stale prices show up.

Everything else in this article is a way of making those five observations repeatable, because a single observation of an assistant is close to worthless. The answers vary from one run to the next for the same question on the same day. A method that does not account for that will report a change every week and never a true one.

Why one answer tells you almost nothing

Ask an assistant the same question three times and you will often get three lists that overlap but do not match. That is how the systems work, not a fault in your question. A measurement that reads one answer as a verdict is a coin flip dressed as a finding: the business that appears once and disappears the next time was never reliably visible, and the one that is missing once may be named in the other two.

Our own measurement asks each question once per reading, of one assistant, Claude, with live web search on, and dates the reading. One answer is a snapshot, so no single answer is read as a verdict. The weight sits on the pattern across the panel's twelve questions and, with monitoring, across months of the same questions, in the way a poll is more reliable than one voter. If you run your own panel, asking each question more than once on the same day shows you the variation directly, at the cost of time.

How to build a question panel that stays comparable

The panel is the set of questions you will ask every month, written the way your buyers type them: casual, abbreviated, specific to your trade and your place. It is built once and then left alone, because a month measured with different questions is not comparable to the month before. For paying clients ours holds twelve questions, set per trade and location and named in the scope.

Write more problem questions than named ones. The named question tells you what an assistant thinks of you; the problem question tells you whether a stranger ever hears your name, which is the measurement that maps to new customers. Avoid questions you would like buyers to ask but they do not; the panel is a record of demand, not an aspiration.

How to record an answer so it can be compared later

1. Keep the answer verbatim, with the date, the assistant and the run number. A summary loses the wording, and the wording is where a changed fact hides.
2. Mark whether your business is mentioned, recommended, or first. Recommended means the assistant put you forward as a choice; mentioned means your name appears in any capacity, including as a caution.
3. List every other business named, in order. Your position is only meaningful against theirs, and a new name appearing in all four assistants at once is the most useful early signal the method produces.
4. Note the facts stated about you: address, phone, price, hours, services. Check them. A wrong fact is a finding on its own, and the fix is on your site, not in the assistant.
5. Note the tone in one word where it is not neutral: praised, cautioned, hedged.

Those verdicts can be turned into a number. Our visibility score is the plainest version: the share of the twelve questions in which the assistant named the business, from 0 to 100, so named in 3 of 12 is 25. Being named first and who else was named are reported beside the score, not folded into it. If you score your own panel, any fixed rule will do, as long as it is written down and never changed between months.

How do you tell a real change from noise?

A change is real when it holds across the runs and across the assistants that showed it, and when you can see what caused it. Three patterns are worth acting on. Your name appears in the problem questions for the first time, in more than one run: something you published became readable and was found. A competitor appears in all four assistants where none of them named it last month: they published something, or were written about, and you should go and read it. A fact about you changed in the answer text: an assistant fetched a page, yours or someone else's, that says something new.

A change is noise when one run of one assistant differs and the others do not, when the position moves by one place in a list whose membership is unchanged, or when the answer is a hedge this month and a list the next. Hedging is common. In our audit of 9,483 small business websites we put one buyer question per business to Claude with web retrieval on, for 9,240 businesses; 5,928 of the answers gave a clearly differentiated list and the rest hedged. An assistant that declines to rank has not demoted you.

The comparison that answers "did they change what they say about us" is therefore not a score against a score. It is this month's verbatim answers laid beside last month's, question by question, read by a person, with the score as the summary of what they found. The score is for the trend; the text is for the reason.

What a first measurement usually looks like

Low, and that is not a verdict on the business. In the same probe, 14.9% of the 9,240 businesses were named at all, and 17.7% of those asked in the answers that gave a differentiated list. That was one assistant, one question per business, on one day, so read it as direction rather than a rate. But it means a first panel that names you in two questions out of twelve is ordinary, and a panel that names you nowhere is the common starting point, not an anomaly.

What the baseline buys you is the ability to see movement. Without it, the work that follows (one identity across your pages, markup, answers written for the questions you are losing, reviews where a machine can read them) produces a feeling that things improved. With it, the second measurement, taken with the same questions, either shows the movement or shows that a particular question type did not move and needs different work.

By hand, or measured for you

By hand, the method above is a spreadsheet and a morning a month: twelve questions, four assistants, three runs each is 144 answers to read and mark. Many owners do exactly that, and the result is a real measurement. The two things that go wrong are drift, where the questions or the weights change and the months stop comparing, and abandonment, where the second month never happens.

The alternatives on this site are a narrower version of that method, run by us. The free visibility check takes a website and an email and returns, within 48 hours, a written reading of whether Claude names you and who it names instead. Monitoring puts a fixed panel of twelve questions to the same assistant, Claude, every month, keeps every answer word for word, and posts each reading on your page with the score, this month against last, and a short note in plain language. It costs $290 a month, and you can cancel in any month. Neither changes what an assistant says. The check tells you what Claude says today, and monitoring tells you whether that changes.

What the measurement cannot tell you

It does not count visits or revenue; an assistant that names you produces a phone call the measurement never sees. It covers only the assistants the panel is put to. Ours is put to one, Claude, and we do not report its answer as ChatGPT's, Gemini's or Perplexity's, because each reads its own index and can answer differently. It answers the questions in the panel, so a buyer who asks something the panel does not contain is invisible to it. And it is only ever comparable to itself: a score from a different panel, a different set of assistants or a different weighting is a different instrument, and two instruments do not make a trend.