Free tool

A/B Test Significance Calculator (Free Tool)

Most declared A/B test winners are noise stopped early. This calculator runs a two-proportion test on your two variants, gives you the p-value and the confidence interval around the difference, and says plainly when the sample is too small to tell them apart.

  • p-value and interval
  • Small samples flagged
  • Nothing is stored

What significance actually tells you

The p-value answers one narrow question: if the two variants were truly identical, how often would you see a difference at least this large by chance alone? A p-value below your threshold means chance is an unconvincing explanation. It does not tell you the size of the improvement, how likely it is to hold, or whether it is worth shipping.

The confidence interval is the more useful number and the one most reports leave out. It is the range of true differences consistent with your data. An interval running from a small decline to a large gain is not a win, however green the significance badge is, because a real decline is still one of the outcomes your data supports.

Peeking is what breaks most tests

Checking the result every day and stopping the moment it crosses the threshold makes false positives far more likely, because a test that fluctuates will eventually cross the line by chance even when nothing is happening. The mathematics behind the p-value assumes one look, at the end, on a sample size you fixed beforehand.

So decide two things before the test starts: how long it runs, and how many conversions each variant needs. Then let it finish. Watching it is fine; stopping it early on what you see is not, unless something is genuinely broken.

Test size, and what is worth testing

The smaller the change you are hoping to detect, the more traffic you need to see it, and the relationship is unforgiving. Most sites do not have the traffic to resolve a small difference in any reasonable period, which is an argument for testing bigger changes rather than for testing more often. A different offer, a different page structure or a different form length moves numbers far enough to be measurable; a button colour usually does not.

When the traffic is not there at all, the honest answer is that testing is the wrong instrument, and the improvements should come from research, session recordings and fixing what is obviously broken. Deciding which of those a site needs is the first step of the conversion rate optimisation service, and it depends on the tracking being right, which is what analytics and tracking covers.

Frequently asked questions

What confidence level should I use?

95% is the usual default and a reasonable one. Raising it to 99% makes false positives rarer but needs more traffic; dropping to 90% finds more results and more of them will be wrong. Pick the level before you run the test, based on how costly a wrong decision would be, and do not change it once you have seen the data.

My result is significant but the interval is wide. Ship it?

Look at the worst end of the interval and decide whether you would still be happy at that outcome. If the lower bound is barely above zero, the safe reading is that B is probably not worse, not that it is much better. Running longer narrows the interval and gives you a decision you can defend.

Can I test more than two variants here?

Not with this calculator, which runs a two-proportion test. Comparing several variants at once raises the chance that one crosses the threshold by luck, so it needs a correction for multiple comparisons. With most sites' traffic, testing two variants properly beats testing four badly.

Ready to talk about your project?

A free 30-minute call, a straight answer about what would move the numbers, and a written proposal within 48 hours if we are a fit.