A/B Test
An A/B test is a controlled experiment where you show two versions of something, A and B, to different segments of your audience at the same time and measure which one performs better against a defined metric. Traffic is split randomly so the two groups are otherwise comparable, and only one variable changes between versions. The result tells you what your audience actually does, not what you assumed they'd do.
Why A/B tests matter
Without a test, a decision rests on whoever argues most confidently in the room, and confident arguments are frequently wrong. An A/B test replaces that with a number: does the new headline, the different button color, or the rewritten product description actually move clicks, sign-ups, or purchases, or does it just look better to the person who made it? Small wording changes can shift a call-to-action's conversion rate more than people expect, and a test is the only way to know which direction it moved and by how much. That number lets you commit budget to what's demonstrably working instead of what someone is confident about, which lowers the cost of being wrong.
How an A/B test works
Start by picking one metric you'll judge the test on, such as click-through rate or conversion rate. Build two versions that differ in exactly one element, for example the headline, and nothing else. Visitors are assigned to A or B at random so the two groups are equivalent going in. Keep the test running until each variant has seen enough traffic to make the result stable, not just until one side happens to be ahead. Then check whether the gap between A and B is large enough that it's unlikely to be noise rather than a real effect. A useful gut check: could you explain the difference to a colleague without hedging, or is it still small enough that it might flip next week.
Common mistakes
The most common mistake is stopping early: calling the test after two days because A is ahead, when the gap is still well within normal daily fluctuation. Too little traffic produces a result that can reverse itself the following week. A second mistake is changing several things in one test, new headline, new color, and new image at once, which means you can't tell which change actually drove the result. Running a test across a holiday or a one-off spike in traffic distorts the numbers in the same way. And a test that won six months ago doesn't automatically still win: audiences, competitors, and context shift, so decisions worth defending get revisited periodically rather than treated as settled forever.
Relation to AI visibility
The same testing instinct applies to how AI systems cite your content, but the mechanics are different from a marketing A/B test. There's no random 50/50 traffic split you can run against ChatGPT or Gemini; instead you publish one version, run the same set of prompts against it over time, and track whether mention or citation rates change, then compare against an earlier version or a rewrite. What's worth testing is grounded, not guesswork: Google has stated plainly that no special markup, schema, or AI-specific files are required for AI Overviews or AI Mode, and that writing separate content "for AI" isn't the lever. What correlates more strongly with getting cited is being mentioned across other sites and sources in the first place, an Ahrefs analysis of roughly 75,000 brands found mention frequency correlates with AI citation rate at about 0.664, close to three times the correlation seen for backlinks. So the more useful comparison is often structural: does a clearer definition, a tighter FAQ, or more specific evidence change how often a page gets picked up, rather than which HTML tag surrounds it.
Example
An online shop selling hiking boots wants more newsletter sign-ups. The button currently reads "Sign up." The team suspects a concrete benefit will outperform a generic label and tests it against "Get free hiking tips." Both versions run in parallel for two weeks, with every other visitor randomly shown one or the other. The benefit-led variant ends up converting noticeably better. Without the test, nobody would know whether the new copy genuinely worked or just sounded better in the meeting where it was proposed. It replaces the old button permanently, on the strength of the numbers rather than a preference.
Common questions
How long does an A/B test need to run?
Long enough for each variant to collect a stable, sufficient sample, and to cover at least one full weekly cycle so weekday and weekend behavior are both represented. As a rough floor, plan on one to two weeks. Don't call the result just because one variant is ahead on day two.
How is it different from a multivariate test?
An A/B test changes exactly one element between two versions. A multivariate test changes several elements at once and measures the combinations between them. Multivariate testing can reveal interaction effects an A/B test would miss, but it needs far more traffic to reach a reliable answer, so A/B remains the simpler default.