Incrementality testing vs A/B testing: the difference, and when to use each
An A/B test compares two versions of something, such as two ads or landing pages, and both groups see advertising. An incrementality test, or lift test, compares a group that sees advertising with one that sees none, measuring sales that would not otherwise have happened.
Both split an audience or market into groups. The difference is what the control receives: another version in an A/B test, nothing in an incrementality test.
A/B testing, defined
- Two versions: each group sees a different ad, headline, landing page or subject line.
- Fast: built into most ad platforms; results can come in days when traffic is high.
- Multivariate tests cross several elements at once and need more traffic.
- The limit: both groups see marketing, so it cannot show whether the activity itself added sales.
Incrementality testing, defined
- Exposed or not: one group sees the channel, a comparable group does not; the difference is lift.
- By person or region: a platform conversion lift study splits people; a geo test splits regions.
- The question: would these sales have happened without the spend?
- On and off over time in one market is simple, and more exposed to seasonality.
The differences, dimension by dimension
Each line: A/B test, then incrementality test.
- Question: which version is better, or whether the activity adds sales at all.
- Control: another version, or no advertising from the tested channel.
- Result: an A/B test always names a winner; a lift test can return zero.
- Scope: inside a channel, or the channel as a whole.
- Time and cost: days and little beyond setup, or several weeks of withheld or added spend.
How to calculate incrementality
Compare the groups over the test period, adjusted for how they compared before. Illustrative figures:
- Lift: exposed minus control, divided by control: 1,100 orders against an adjusted 1,000 is 10%.
- Incremental conversions: the difference, here 100 orders.
- Incremental return on ad spend: incremental revenue divided by spend, usually below the platform's figure.
- Uncertainty: report how certain the estimate is and the smallest effect the test was designed to detect.
Why a winning A/B test can still lose money
- A winner either way: if neither version adds sales, one still wins and keeps the budget.
- Retargeting: an A/B test picks the better creative, not whether those people would have bought anyway.
- The fix: a holdout on the retargeting audience.
How long an incrementality test needs
Across 123 of our own geo experiments:
- Under two weeks: 27% of 11 tests reached significance.
- Four to six weeks: 71% of 34 tests.
- Past six weeks the rate stops improving; spend level separated nothing.
- A/B tests read sooner: two versions often differ more than a whole channel's effect.
What each test needs to be valid
- A/B test: evenly split groups over the same period, enough traffic, one change at a time.
- Incrementality test: a comparable control, enough weeks, no overlapping launches or promotions.
- Mistakes in both: stopping once it looks significant, changing budgets or creatives mid-test, picking the metric afterwards.
What neither test does
- One thing, one period: results date as spend and seasons change.
- One spend level: for other budgets, repeat the test or calibrate a marketing mix model with it.
- Volume: with few conversions or regions, tests are underpowered and results often unreliable.
How to choose
- Which version works better: run an A/B test.
- Whether a channel adds sales at all: an incrementality test, before scaling or when returns look too good.
- A programme: lift tests pick the channels, A/B tests improve them, a marketing mix model sets the split.
Cassandra is an AI marketing brain for growth teams. It measures what each channel adds with a Bayesian marketing mix model, checked by live geo experiments. Start for free with your website and data, or book a demo.
Adding incrementality tests to an A/B programme
- Start with the largest channel, or the one whose reported returns are least trusted.
- Protect the calendar: agree a window of several weeks with the teams running promotions.
- Rhythm: one incrementality test a quarter on the largest channels, A/B tests continuously.
Questions
What does incrementality testing mean?
Measuring the extra sales an activity produces by comparing a group that saw it with a comparable group that did not. The difference is the incremental effect.
How do I calculate incrementality?
Subtract the control's outcome from the exposed group's, adjusted for how they compared before. Lift divides that by the control outcome; incremental return divides incremental revenue by spend.
Is a lift test the same as an A/B test?
No. An A/B test compares two versions, both with advertising. A lift test compares advertising with none.
How long does an incrementality test need to run?
Across 123 of our own geo experiments, 27% of tests under two weeks reached significance against 71% of tests running four to six weeks.
Can an A/B test measure incrementality?
Only if one arm receives no advertising, which makes it an incrementality test.