How a geo holdout test works
You divide your market into regions that behave alike — similar population, similar past sales, similar seasonal pattern. One group keeps running ads as normal. The matched group goes dark for a fixed window. Because nothing else changed between them, the gap in sales that opens up is the part the advertising was actually causing.
The matching is the real work. Regions are paired on their sales history before anything is switched off, and the pairing is written down in advance so nobody is tempted to swap a region out once the results start to look inconvenient. Ad platforms let you target down to city or district level, which is fine enough for most markets, though a small country with one dominant city gives you fewer usable pairs than a spread-out one.
The answer never comes from the ad platform’s own reporting. It comes from your own sales records — orders, bookings, enquiries — compared across the two groups over identical dates.
Why geo holdout tests matter
Every ad platform reports the conversions it feels able to claim, and none of them can tell you what would have happened if the ad had never appeared. A share of the sales credited to retargeting would have arrived anyway, from people already on their way to buying. A geo holdout answers the harder question: is this spend creating demand, or collecting credit for it?
It also settles arguments that reporting alone cannot. When two channels both claim the same sale, a holdout removes one of them from part of the map and lets your sales figures decide who was right.
Common mistakes with geo holdout tests
Stopping early is the usual one. A test read after a few days is reading noise, and the pull to call it the moment the numbers look favourable is strong. Set the end date before you begin and honour it.
The second mistake is a holdout too small to say anything. If the paused regions produce only a trickle of orders, the difference you measure will be swamped by ordinary week-to-week variation — a question of statistical significance, not of effort. The third is contamination: running a promotion, a press feature or an unrelated campaign in one group and not the other, which quietly ruins the comparison without anyone noticing.
How to act on it
Test the channel or tactic you are least sure about, not the one you already trust. Agree in advance what result would make you cut the spend and what would make you raise it, because a test with no decision attached is an expensive way to be curious.
Run it long enough to cover a full buying cycle, keep everything else steady, and accept that a holdout costs you the sales you deliberately gave up. That cost is the price of knowing. Repeat it occasionally rather than once, since the answer moves as your market, your competitors and your creative change.
