Analytics and Tracking

Incrementality Test

Also called Lift test, holdout test, geo experiment

An experiment that withholds ads from a comparable group so the sales a channel truly caused can be measured.

Quick facts: Incrementality Test

Category
Analytics and Tracking
Also called
Lift test, holdout test, geo experiment
Level
Advanced
Affects
Budget decisions, true channel value, trust in attribution
Where to see it
Google Ads experiments, Meta conversion lift, geo splits in GA4 or your own sales data
In this article4
  1. How an incrementality test works
  2. Why incrementality matters
  3. Where incrementality tests go wrong
  4. How to run one properly

How an incrementality test works

You split your audience or your map into two comparable halves. One keeps seeing the advertising; the other is deliberately left alone. Everything else — season, pricing, promotions, other channels — applies to both. After the test period you compare outcomes. The difference between the two groups is the part the advertising caused, and everything else was going to happen anyway.

The split can be done in several ways. A geo test switches a channel off in some districts or cities and leaves it running in others. An audience holdout keeps a randomly chosen slice of a remarketing list out of the campaign. Google and Meta both offer built-in lift studies that hold back a control group inside their own systems, which is convenient but keeps the measurement inside the platform being tested.

Why incrementality matters

Attribution reports answer a question about credit; an experiment answers a question about cause. They are not the same, and the difference is largest exactly where the money is. Branded search, remarketing and catalogue ads all sit in front of people who already intended to buy, so they collect a great deal of reported credit for sales that would have arrived without them.

An incrementality test is the only method that settles that argument with evidence rather than opinion. It is also the only measurement that survives cookie loss, consent refusal and closed platform reporting, because it never needs to follow an individual person from click to purchase.

Where incrementality tests go wrong

Most failed tests are too small or too short. If your weekly order count is low, the natural noise between two groups can be larger than the effect you are hunting for, and the result means nothing. Working out the required duration before you start, rather than after, is the difference between an experiment and an anecdote.

Contamination is the other frequent fault. Control regions see the ads anyway because targeting is loose, staff run a promotion in the test area, or the holdout group is small enough that ordinary week-to-week variation swamps the signal. Stopping the moment the result looks good is a third, and it reliably produces a flattering answer that does not repeat.

How to run one properly

Decide the question first: not “does Meta work” but “if we switched off catalogue remarketing for a month, how many fewer orders would we take?” Choose groups that behaved alike before the test, agree the test length in advance, and change nothing else while it runs. Read the outcome against a plain arithmetic check on whether the difference is large enough to be real.

In a market the size of Nepal, clean geo splits are hard: a large share of demand sits in the Kathmandu valley, so removing it leaves an unbalanced control. Audience holdouts and simple on-off periods on a single campaign are usually more practical, and are worth running before committing a serious budget. Where volumes are genuinely too small, say so and rely on consistent tracking plus business-level results instead of pretending an underpowered test proved something.

Do and do not

Do

  • Decide the question and test length before switching anything off
  • Choose test and control groups that behaved alike beforehand
  • Keep promotions, pricing and other channels unchanged during the test

Do not

  • Stop the test early because the result looks good
  • Run a test when order volumes are too low to read
  • Let the control group see the ads through loose targeting

Questions people ask about this

How long should an incrementality test run?

Long enough to collect a meaningful number of conversions in both groups and to cover at least one full sales cycle, including the lag between first click and purchase. Set the length before starting and hold to it. Short tests on low order volumes produce results that swing with ordinary weekly noise rather than with the advertising.

Is a platform lift study the same as an incrementality test?

It is one kind, run inside the platform being measured. The mechanism is sound: a control group is held back and outcomes compared. The limitation is independence, since the platform defines the groups, the conversion and the window. Use it for direction, and run a geo or holdout test outside the platform when a large budget depends on the answer.

What if my business is too small to run a proper test?

Then be honest about it rather than running an underpowered experiment. Simple on-off periods on one campaign, watched against your own order records over several months, will still reveal an obvious effect. Combine that with a question on the thank-you page asking how customers found you, and judge the whole programme on blended results.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.