Marketing mix modeling vs incrementality testing: the differences, and how to use them together
Marketing mix modeling estimates every channel's contribution from historical spend and sales, and suits budget allocation. Incrementality testing runs a controlled experiment, such as a geo test, to measure one channel's causal effect; most teams use both, with tests calibrating the model.
A model covers the whole mix continuously, with stated uncertainty. A test measures one channel's cause and effect directly in a window, because it changes spend on purpose.
Marketing mix modeling, defined
- Input: weekly spend by channel, the outcome, and seasonality, pricing, promotions and the economy.
- Output: each channel's contribution, with delayed effects and diminishing returns, online and offline.
- Answers portfolio questions: the budget split, where the next unit returns most, what a change will do.
- Limit: if spend rose when demand rose, the data alone cannot tell which drove the other.
Incrementality testing, defined
- Geo test: spend in one channel rises or is held back in some regions; matched regions carry on.
- The difference in outcomes between the groups is the incremental effect.
- Other designs split audiences, or switch spend on and off; all need a comparison group.
- Decide upfront: regions or audiences, length, and the smallest detectable effect, so a null result can be read.
The differences, dimension by dimension
Model, then test:
- Scope: every channel at once, versus one channel or campaign.
- Method: infers from historical variation, versus creating it.
- Answer: an estimate with a range, versus a measured effect for its conditions.
- Timing: runs on existing history today, versus a window of several weeks ahead.
- Scaling: shows diminishing returns, versus one measured point.
Why a model benefits from a test
- A model can mistake timing for effect when spend and demand rise together; a test's spend change is independent of demand.
- The model must then reproduce the tested result, which can improve the other channels' estimates too.
- So a few well-chosen tests improve the whole model.
Why a test benefits from a model
- One question each: testing eight or ten channels in turn outlasts a planning cycle.
- Results date: a model carries a result forward and shows other spend levels.
- What to test next: the least certain estimates where the most budget is at stake.
How long an incrementality test needs to run
Across 123 of our own geo experiments:
- Under two weeks: 27% reached significance (11 tests).
- Four to six weeks: 71% (34 tests). Past six weeks the rate stops improving; spend level separated nothing.
- Protect the window: no overlapping launches or budget changes in the test regions.
- Comparison regions must have moved in step with the test regions, chosen before results.
When an incrementality test on its own is the better choice
- One channel dominates the budget and one decision is pending.
- A disputed figure, such as a questioned platform return, needs a direct answer.
- Too little history for a model: a test creates its own variation.
When a marketing mix model on its own is the better choice
- No valid comparison group, such as one market with no regional structure.
- Allocation across the whole mix, which depends on channel interactions and diminishing returns.
- A decision due before a test window could finish.
What neither method does
- Daily optimisation inside a channel needs other tools.
- Thin data: a model returns a wide range; a too-short test returns no readable result.
- Conditions change: re-run a test when spend level, season or the channel's role shifts substantially.
How to choose
Start with the method that answers the decision already due.
- Budget split for a planning cycle: a marketing mix model.
- One channel's value in question: an incrementality test.
- A lasting programme: both; the model allocates and picks tests, tests calibrate the model.
Cassandra is an AI marketing brain for growth teams. It measures each channel with a Bayesian marketing mix model checked by live geo experiments, and does the work within the team's rules. Start for free with your website and data.
Starting a combined programme
- Data work: a clean history of spend and outcomes feeds the model and the choice of test regions.
- Sequence: fit a model, test its least certain estimate, feed the result back.
- Record every test: channel, regions, window, spend change, result.
Questions
What are incrementality tests in marketing?
Experiments that measure the extra sales a channel produces by changing its spend for one group, such as a set of regions, and comparing the outcome with a similar group left unchanged.
Is incrementality testing better than marketing mix modeling?
They answer different questions: a test measures one channel directly inside a window, a model estimates every channel continuously. Most programmes use tests to calibrate the model.
How long does an incrementality test need to run?
Across 123 of our own geo experiments, 27% of tests under two weeks reached significance (11 tests), against 71% of tests running four to six weeks (34 tests). Past six weeks the rate stops improving.
Do you need to test every channel?
No. A measured effect on one channel becomes a constraint the model has to satisfy, which can improve the estimates for the rest of the mix.
Can a test replace a marketing mix model?
For one channel and one decision, often yes. For allocation across the whole mix, no: sequential tests take longer than a planning cycle and do not show how channels interact.