Geo experiments vs holdout tests: how each design works, and which one fits your business
A geo experiment changes spend in some regions and compares them with similar regions left unchanged. A holdout test withholds advertising from a group and compares it with a group that saw it; a geo holdout is both, pausing spend in the test regions.
The useful distinction is the unit split: a geo design splits regions and needs sales by region; an audience holdout splits people and needs exposure linked to purchase.
Geo experiments, defined
- Regions: cities, designated market areas or postcodes. Spend changes in test regions; matched regions carry on.
- Effect: the outcome difference, against how the groups tracked each other before the test.
- Two directions: a scale-up test adds budget; a geo holdout, or go-dark test, pauses or reduces it.
- Matched markets pair each test region with one or a few similar regions.
- Synthetic control blends many untouched regions; open-source tools such as Meta's GeoLift use it.
Holdout tests, defined
- Withholds advertising from a group defined by region (geo holdout) or by person (audience holdout).
- Platform studies: ad platforms offer audience holdouts as built-in studies, often called conversion lift or brand lift.
- Needs the platform to link each person's exposure to their later purchase.
The differences, dimension by dimension
Geo experiment, then audience holdout:
- Unit: regions, versus people.
- Needs: sales by region and enough regions, versus exposure linked to purchase per person.
- Channels: offline sales, TV, connected TV, audio, out-of-home, versus addressable digital.
- Who runs it: usually the advertiser or an independent party, versus the platform being evaluated.
- Contamination: people crossing region boundaries, versus reach through other channels or devices.
The question that decides which design you can use
Can each person's exposure be linked to their purchase, reliably, across the whole path?
- If yes, an audience holdout is available and often cheaper, because the platform handles setup.
- If no, a geo design works, because it only needs sales by region.
- The link breaks with in-store purchases, connected TV and audio; shared devices and consent limits weaken it.
How long a test needs to run
- Across 123 of our own geo experiments, 27% of tests under two weeks reached significance.
- Four to six weeks: 71% did.
- Spend level separated nothing.
- Plan four to six weeks with no launches, promotions or budget changes in the test regions.
What a geo test costs in revenue
- Across 15 of our own geo experiments that returned a clear read, the median share of revenue exposed was 0.7%.
- Nine in ten exposed under 2.7%.
- A scale-up test adds spend, so it puts no existing revenue at risk.
Who reads the result
- Platform-run holdouts are designed, run and reported by the party whose budget is evaluated.
- So many teams also test their largest channels independently.
- Write the design down before the start: regions and why, length, smallest detectable effect.
- Detectable effect: a test sized for a large effect can report none for a moderate one.
What neither design does
- One channel, spend level and period per test; repeat when conditions change.
- The whole mix: test results calibrate a marketing mix model covering the rest.
- No comparison group: a single-market company with no regional structure or addressable audience cannot run either cleanly.
How to choose
- Online purchase linked person by person, speed over independence: an audience holdout is quicker.
- Offline purchase, TV, audio or out-of-home, or no identity join: a geo experiment, given regional structure.
- A large budget at stake: a design someone other than the channel's seller can check.
Cassandra is an AI marketing brain for growth teams. It measures each channel with a Bayesian marketing mix model checked by live geo experiments, and does the work within the team's rules. Start for free with your website and data.
Moving from platform tests to independent tests
- Platform lift studies are a common start, subject to each platform's eligibility requirements.
- Go independent with the largest channel, where the stakes justify a result the platform does not grade.
- Groundwork: a clean sales history by region, which a marketing mix model also needs.
Questions
What is a geo holdout test?
A geo experiment that pauses or reduces spend in a group of test regions and compares their sales with similar regions where spend continued.
What is a holdout test?
A test that withholds advertising from a group, defined by region or by person, and compares that group's outcome with a group that could see the advertising. The difference is the incremental effect.
What are geo experiments?
Tests that change spend in one group of regions and compare the outcome with a matched group left unchanged. They need no user-level tracking, so they work for offline and broadcast channels.
How much revenue does a geo test put at risk?
Across 15 of our own geo experiments that returned a clear read, the median share of revenue exposed was 0.7% and nine in ten exposed under 2.7%. A scale-up test adds spend, so it puts no existing revenue at risk.
Are platform-run lift studies reliable?
They are useful and quick. They are designed, run and reported by the platform whose budget is being evaluated, so many teams confirm results for their largest channels with an independent test.