Incrementality testing: how to find out what your ads really cause
Incrementality testing measures how many sales happened because of an ad, by comparing a group that saw the ads with a similar group that did not. Marketing attribution assigns credit for sales that were tracked, and marketing mix modeling estimates each channel's contribution from historical spend and revenue. Attribution runs the account day to day, incrementality tests check the big budget decisions, and marketing mix modeling fits companies spending across many channels with at least two years of data.

Three measurement methods, three different questions
Most arguments about measurement come from asking one tool to answer another tool’s question.
Marketing attribution takes conversions that were tracked and assigns credit to the clicks and views that came before them. It works at the level of campaigns, ad sets, keywords and ads, and it updates daily. Its weakness is that it only counts what it can see, and it assumes that a tracked touch before a sale helped cause the sale.
Incrementality testing runs an experiment. One group gets the ads and a comparable group does not. The difference in outcomes is what the ads caused. It is the closest thing marketing has to a controlled trial. Its weakness is that each test answers one question, at one point in time, and costs some revenue while it runs.
Marketing mix modeling (MMM) is a statistical model built on history: weekly spend by channel, revenue, price, promotions, seasonality. It estimates how much each channel contributed and where returns flatten. It needs no user-level data, so it covers TV, podcasts, influencers and retail. Its weakness is that it needs a lot of data and it finds correlations, so it should be checked against experiments.
| Attribution | Incrementality test | Marketing mix modeling | |
|---|---|---|---|
| Question it answers | Which ads got credit for tracked sales | Did this channel cause sales | How much did each channel contribute over time |
| Data needed | Conversion tracking | Regions or audiences you can split | Two or more years of weekly spend and revenue |
| Level of detail | Campaign, ad, keyword | One channel or tactic per test | Channel, sometimes tactic |
| Speed | Daily | Three to eight weeks per test | Weeks to build, then refreshed monthly or quarterly |
| Main risk | Credits sales that would have happened anyway | Noisy result if groups are small or mismatched | Confuses correlation with cause |
The three work together. Attribution steers the account between tests. Tests correct attribution. A mix model, when you have one, is calibrated with test results.
Which method fits your budget
These are rules of thumb. Purchase volume and the number of channels matter more than the spend figure itself.
- Under $50,000 a month, one or two channels. Use platform attribution, clean conversion tracking and one blended number such as marketing efficiency ratio. Our post on MER vs ROAS explains how to use it. Formal tests are usually too noisy at this size.
- $50,000 to $250,000 a month. Keep attribution for daily work and add one or two incrementality tests a year on the largest line items. Typical first tests: Meta prospecting, branded search, retargeting and Performance Max.
- Above $250,000 a month, or four or more channels including offline. A mix model becomes worth building. Open-source tools such as Google’s Meridian and Meta’s Robyn have lowered the cost, but the model still needs clean data and someone who can question its output. Keep running experiments to calibrate it.
Both Meta and Google offer conversion lift studies inside their platforms. These split users into exposed and unexposed groups and are a good option when your account qualifies. A geo holdout has one advantage: you run it yourself, on your own revenue data, with no dependence on the platform’s tracking.
How to run a geo holdout test
Step 1: Write the question and the decision
A test is worth running only if the result changes a decision. Write it down before you start, for example: “If Meta prospecting returns less than $1.50 of incremental revenue per dollar, we cut its budget by 30%. If it returns more than $2.50, we raise it.”
Step 2: Choose and match regions
Use regions the ad platforms can target and your store can report on: states or metro areas in the US, countries or regions elsewhere. Build two groups whose revenue moved together over the past 8 to 12 weeks. Avoid putting your single largest market in the holdout, and avoid regions with a store opening, a local promotion or a stock problem.
Step 3: Fix the rules before launch
Set the length (three to six weeks is common), the primary metric (total revenue or new customers from your backend, never platform-reported conversions), and the analysis method. Do not change other marketing in either group during the test.
Step 4: Turn the ads off in the holdout regions
Exclude the holdout regions from the campaigns being tested. Leave everything else as it was. Budgets in the control regions stay at their normal level per region, so total spend falls for the duration.
Step 5: Compare actual with expected
Use the pre-test relationship between the groups to predict what the holdout regions would have sold with ads on. The gap between that prediction and actual sales is the incremental revenue.
A worked example (hypothetical)
This example uses invented numbers to show the arithmetic.
A brand spends $150,000 a month on Meta. In the 12 weeks before the test, the holdout regions consistently produced revenue equal to 25% of the control regions’ revenue. On a normal month, the holdout regions receive $30,000 of the Meta budget and the control regions $120,000.
Meta ads are switched off in the holdout regions for four weeks.
| Item | Value |
|---|---|
| Control regions revenue during test | $2,000,000 |
| Expected holdout revenue (25% of control) | $500,000 |
| Actual holdout revenue | $440,000 |
| Incremental revenue lost by turning ads off | $60,000 |
| Meta spend withheld in holdout regions | $30,000 |
| Incremental ROAS ($60,000 / $30,000) | 2.0 |
| Meta-reported ROAS for the same regions before the test | 3.5 |
Meta’s own reporting would have credited $30,000 of spend with $105,000 of revenue (30,000 x 3.5). The test found $60,000. The ratio, 60,000 / 105,000, is about 0.57. For planning, this brand can multiply Meta-reported revenue by roughly 0.57 to estimate incremental revenue, until the next test updates the figure.
Whether an incremental ROAS of 2.0 is good depends on margin. With a 60% contribution margin, each dollar of ad spend returns $1.20 of margin, so the channel pays for itself. With a 40% margin it returns $0.80 and loses money on the first order. Our post on what is a good ROAS works through the break-even math.
One caution about the example: a single test produces an estimate with a range around it. If normal week-to-week noise between the two groups is 5%, a 12% drop is a clear signal. A 3% drop would not be.
Common mistakes
- Testing with too little volume. If the holdout regions produce a few dozen orders a week, random variation will swamp the effect. Use larger regions or a longer test.
- Mismatched groups. Regions that differ in seasonality, store presence or customer mix will drift apart on their own. Check the pre-period fit before launch.
- Stopping early. Looking at results every day and ending the test when the number looks good produces false findings. Fix the end date in advance.
- Ignoring delayed purchases. People who saw an ad in week four may buy in week six. Keep measuring for one to two weeks after ads come back on.
- Measuring with platform data. The point of the test is to check the platform. Use backend revenue.
- Changing other things mid-test. A price change, an email blast to one group or a new campaign in the control regions ruins the comparison.
- Treating one result as permanent. Incrementality changes with spend level, creative and season. Retest the largest channels at least once a year.
- Testing without a decision attached. A result that nobody acts on cost revenue and bought nothing.
How incrementality changes budget decisions
The usual finding is that channels differ more in incremental return than in attributed return. Branded search and retargeting often look best in platform reports and less strong in tests, because they reach people who were already close to buying. Prospecting often looks worse in reports and better in tests. The size of the gap is specific to each business, which is the reason to test instead of borrowing someone else’s number.
Once you have an incremental factor per channel, apply it to marginal decisions. When you scale ad spend, the next dollar should go to the channel with the best incremental return at the margin, which is rarely the one with the best average ROAS in the dashboard.
What to do next
List your three largest line items by spend and ask which one you have the least evidence for. That is the first test. Check that backend revenue can be reported by region, pick matched groups, and write the decision rule before launch.
We design and read these tests as part of marketing analytics, and for managed accounts they are included in PPC management. Daniel plans the test and the budget decision that follows; our engineering partner builds the regional revenue reporting it depends on.