What power measures
Power is a property of the test, not of the ad. Before you compare two creatives, three things are already fixed: how different the two genuinely are, how much the daily numbers bounce around on their own, and how much traffic each version will receive. Power is what those three add up to — the probability that the test will notice a real difference rather than shrug and report nothing.
The uncomfortable part is that the smaller the true difference, the more data it takes to see it. A creative that is dramatically cheaper announces itself quickly. One that is only mildly better needs a great deal of traffic before the gap separates from noise. Volume is the lever you actually control, which is why sample size and test duration are the first things to settle.
Why power matters
Most small accounts run tests that could never have worked. An underpowered test comes back inconclusive whatever the truth is, so the team decides the creatives are equivalent, keeps the incumbent, and repeats the same exercise next month. Nothing is learned, but budget is spent learning it.
Low power also produces the opposite failure, which is more dangerous because it looks like success. When a test is starved of data, the only results that cross the statistical significance line are the ones where random noise happened to push in the same direction as the real effect. The measured winner is therefore overstated, and the performance you scale to is not the performance you saw.
Common mistakes with power
The first is testing on the rarest event in the account. Purchases are what you care about, but they happen far less often than clicks or video views, so a purchase-level test needs a much larger budget than a click-level one. The metric you judge on decides how long you must wait.
The second is splitting the budget too many ways. Four variants running at once does not give you four times the learning; it gives you four starved cells and no conclusion. On the smaller audiences and modest budgets that are normal in Nepal, two well-separated options usually beat a wide bracket.
The third is checking every morning and stopping the moment one side looks ahead. Repeated peeking finds a false lead sooner or later on almost any pair of identical ads.
How to act on it
Decide before launch what difference would actually change your decision, then work out how much volume that needs and whether you can afford it. If the required volume is out of reach, do not run the test — that is a legitimate answer, and it saves the budget for something that can be resolved.
When volume is genuinely too thin, test bigger things. Comparing two distinct ideas needs far less data than comparing two shades of a button, because the effect being measured is larger. For the rest, judge creatives on cheaper upstream signals such as hook rate and cost per click, plus honest commercial judgement, and accept that you are ranking rather than proving.
