Setting up prompt monitoring: what to ask, how often, and what to record

What is prompt monitoring?

Prompt monitoring is the measurement side of generative engine optimization. You write down the questions a buyer types into ChatGPT, Claude, Gemini or Perplexity when they are looking for a business like yours, you put those exact questions to the assistants on a schedule, and you keep a record of every answer. It is brand monitoring for a channel that publishes nothing: the answer exists once, for one person, and is gone unless you asked and kept it.

A prompt is the whole text the assistant works from, and for a buyer that is a question in plain words: "who does emergency furnace repair in Duluth". Monitoring a prompt means asking that question the way a buyer would, not the way a marketer would. No brand names the buyer would not know, no trade jargon, no keyword lists.

Three things get recorded for each answer: whether your business is named at all, where it sits when it is (first, in the list, or in an aside), and who else was named. The third one matters as much as the first. An answer names two or three businesses, not ten, so your AI visibility is a share of a small space, and the other names are the people you are sharing it with.

Why does the panel have to stay fixed?

Because a reading only means something against the last one, and two readings compare only if the questions behind them are identical. Change one word in a question and you have started a new series, not continued the old one. "Best plumber in Reno" and "good plumber near me in Reno" pull different answers, and a rise from one to the other is not a rise.

So the panel is written once, set for your trade and your location, and then left alone. The first run is your baseline, and every later run is read against it. When you want to know about a new question, you add it as a new line with its own start date rather than swapping it for one that has history.

The temptation to edit comes after a bad month. The question that never names you looks wrong; surely buyers do not ask it that way. Sometimes that is true, and the fix is to retire the line and note why. What you cannot do is reword it until the answer improves, because then the improvement belongs to the wording, not to the site.

Why is one run a coin flip?

An assistant does not give the same answer twice. Ask the same question three times in a row and you can get three lists that overlap but do not match, with a different name first each time. A single run tells you what happened once. It cannot tell you whether that was usual.

Running each question three times in a row is one answer to that. Three runs do not remove the variation; they let you see it. Being named in three runs of three is a different state from being named in one of three, and a single run cannot tell those two states apart. Our own measurement answers it another way. It asks each question once per reading, of one assistant, Claude, and dates the reading, so no single answer is read as a verdict; the weight sits on the pattern across the panel's twelve questions and, with monitoring, across months of the same questions.

We have published one large single-run number ourselves, and we label it accordingly. In our audit of 9,483 small business websites (40 trades, 90 metros, September 2026), one buyer question per business went to Claude with web retrieval on, once, and 14.9% of the businesses were named. That is a Claude figure from one question on one day, and we call it directional rather than a rate for the reason above. Your monitoring should hold itself to the same standard.

How to build the panel

A panel is twelve questions for a paying client; for your own first pass, six is enough to learn the shape. Each question is written the way a buyer types it, with the place named the way a local would name it, and each belongs to one of three types.

Split the panel roughly in half between problem questions and the other two types, because the problem question is where new customers come from and the other two tell you why. Write the questions in a file with the date each was added. That file is the panel; a memory of it is not.

How to record an answer, and what to do with the numbers

Every answer gets the same five fields, filled in by a person who read the whole thing. The verbatim copy is the one people skip, and the one you will want in three months when a verdict looks wrong.

1. Named? Your business appears anywhere in the answer, under any spelling of its name. Yes or no.
2. Recommended? The assistant puts you forward as a suitable choice, rather than mentioning you in passing or as a caution.
3. First? You are the first business named. In a list, position one; in prose, the first name the reader meets.
4. Who else? Every other business named, in order. This is the field that turns a score into a picture of your share of voice.
5. Verbatim. The full answer, pasted, with the date, the assistant and the run number. Never a summary.

date: YYYY-MM-DD
assistant: ChatGPT
run: 2 of 3
question: who repairs a leaking water heater in Mesa the same day
named: yes
recommended: yes
first: no (second of three)
others: [business A], [business B]
answer: (pasted in full, unedited)

The numbers come from the fields. You can weight them, so that a mention counts, a recommendation counts more and first place counts most, and combine the verdicts across all questions, assistants and runs into one number from 0 to 100. The reason to weight at all is that being named in an aside and being the first recommendation are not the same event and should not add up the same; write the weights down and keep them. Our own score, which our methodology page sets out, does not weight: it is the share of the twelve questions in which the assistant named the business, from 0 to 100, and being named first and who else was named are reported beside it. Take the reading monthly, on the same questions, and keep the trend, your share of all recommendations, and the breakdown by question type next to the headline score.

Doing it by hand, or having it done

By hand, the arithmetic is the constraint. Twelve questions, four assistants and three runs is 144 answers a month, each read by a person and filed with its five fields. Done with care, that is most of a working day every month, by someone who will not reword a question after a bad result. If you have one location, one assistant you care about and that discipline, a spreadsheet is enough and you should start there.

What we sell is a narrower version run for you: monitoring at $290 a month, which you can cancel in any month. The twelve questions are written for your trade and town and put to one assistant, Claude, every month. Every answer is kept word for word. Each reading is posted on your page with the score, this month against last, and a short note in plain language, and we email you when it is. It is the right choice when the reading has to be defensible to someone else, or when the person who would do it by hand is also the person who fixes the site and cannot be both.

Either way, the reading changes nothing by itself. It tells you which questions never name you and who is named instead, and the work of changing that is on your site: an identity a machine can pin down, answers written under the questions, reviews it can read. Monitoring is how you find out whether that work did anything.