Geo Holdout Test

A geo holdout test measures incremental lift by turning ads off in a set of regions and leaving them on in matched regions, then reading the difference in installs or revenue between the two.

You split the country into comparable regions, keep spend running in the test group, and go dark in a matched control group. Everything else stays the same. After a few weeks the gap between the two groups, adjusted for their pre-test baseline, is the lift the ads actually caused. Unlike attribution, nobody claims credit here; the counterfactual is the region where you stopped spending.

Suppose you suspect your branded search and retargeting are just billing installs that would have happened anyway. You pick ten regions that historically track your national installs closely, shut those campaigns off there, and keep them live everywhere else. Four weeks later the held-out regions show installs down only three percent against control. Most of what those campaigns were charging you for was not incremental, and you can cut the budget without losing the volume.

The whole test lives or dies on how well control matches test. If your control regions were already trending differently, or the test window catches a regional holiday or a competitor's local push, the lift number is noise wearing a lab coat. Match on pre-period behavior, run long enough to clear weekly cycles, and do not read a result off a window shorter than your typical conversion lag.

Turn it off where you can afford to, and the drop is the only honest number you own.