How a holdout test works
A holdout test starts by dividing an audience or a market into two groups that look alike. One group carries on seeing the advertising as normal. The other — the holdout — is deliberately cut off from it. You then compare what each group did. The difference between them is the part of the outcome the advertising actually caused, rather than the part that would have happened anyway.
The division can be built in several places. Some ad platforms can hold a share of an audience back automatically. Otherwise the split is geographic: whole districts or cities are switched off while comparable ones keep running, which is the practical route for most businesses. What matters is that the two groups behaved similarly before the test began, and that nothing else changed for one of them during it.
Why holdout tests matter
Every reporting platform counts the conversions it can claim, and remarketing is the clearest illustration of the problem: much of that audience was coming back regardless. A conversion credited to an ad is not proof the ad produced it. A holdout is the only common method that answers the harder question — what would have happened with no advertising at all — because it creates that situation and measures it directly. That makes it the honest way to settle a budget argument about a brand campaign, a retargeting audience or a channel that reports well and cannot be checked. It is closely related to a geo experiment, which is a holdout run at the level of regions.
Where holdout tests go wrong
Holding back too small a group is the usual failure. The result then swings on a handful of conversions and tells you nothing you can bank on. Running the test too briefly is the same problem in another form, particularly where the purchase takes weeks of consideration.
Contamination is the subtler one. A held-back region still sees your social posts, your email, your signage and word of mouth from the region that is being advertised to, so the gap you measure is muddied. There is also a real cost: the holdout group is demand you chose not to pursue. That is the price of the answer, and it should be a deliberate decision rather than a surprise at the end of the month.
How to run one
Decide the question first and the design second. Name the single outcome that will settle it, work out how long the group needs to stay dark for the answer to be readable, and write down in advance what result would change your spending. Then keep everything else steady: same creative, same pricing, same other channels.
Run it long enough to cover the normal gap between first contact and purchase, and be honest afterwards. A holdout that comes back inconclusive is a genuine finding, not a failed test — it usually means the effect is smaller than the design could detect, which is exactly what you want to know before scaling the budget. Feed the answer into how you value the channel from then on, and repeat the test when the market or the offer changes materially.