A/B Test
An A/B test is a controlled experiment in which two versions of one thing are shown to randomly split groups and a single outcome is measured for each, so that a difference in the outcome can be attributed to the difference between the versions.
What makes it a test rather than a change
Four conditions. Assignment is random, so the two groups differ only in the variant they saw. One thing changes between the variants, so a difference in outcome has one explanation. The outcome is named before the test starts. And the test runs until enough people have passed through for the difference to be larger than chance would produce. Drop any one of these and you have a change followed by a guess about what it did.
Why an AI answer is hard to A/B test
A web page can split its visitors. An assistant cannot be split: one crawler fetches one version of the page, and the answer it writes is composed from whatever it fetched. There is no traffic to divide and no way to show version B to half the assistants. The controlled evidence that exists comes from the research side. Aggarwal et al., “GEO: Generative Engine Optimization” (KDD 2024), altered source pages, adding statistics, quotations and citations among other edits, and measured how visible each source became in generated answers on a fixed benchmark. That is an experiment with a control, run over a query set, not on a single business.
For one business the practical substitute is a before-and-after reading with the same instrument: the same question panel, put to the same assistants, before a change and at intervals after it. It does not isolate cause the way random assignment does, and the reading moves for reasons that have nothing to do with the site. The methodology page describes how the panel is held constant so that at least the instrument is not one of those reasons.
A common question
Can a business test a page change against an assistant the way it tests visitors?
No, because there is no traffic to split. One crawler fetches one version of the page, and the answer is composed from whatever that fetch returned, so there is no way to serve a second variant to half the assistants. The practical substitute is a before-and-after reading taken with the same instrument: the same questions, put to the same assistants, before a change and again at intervals afterwards. That does not isolate cause the way random assignment would, and the reading moves for reasons that have nothing to do with the site.
Related terms
Sources
The controlled study of source-side changes: Aggarwal et al., “GEO: Generative Engine Optimization” (KDD 2024). The panel method is this site’s own, on the methodology page.