What statistical significance measures
Run the same page against itself, showing identical content to two random halves of your traffic, and the two halves will still produce different conversion rates. People arrive in different moods, on different days, from different sources. Some difference is always there, and it means nothing.
Statistical significance is the calculation that separates that ordinary variation from a real effect. It asks how likely a gap this large would be if the two versions were genuinely identical. When that likelihood is small enough, the result is called significant, and the convention is to fix the threshold before the test starts rather than after seeing the numbers.
Three things drive it: how many people saw each version, how often they converted, and how large the gap is. A big difference shows up quickly. A small one may need more traffic than your site will ever produce in a sensible period.
Why statistical significance matters
Without it, testing quietly turns into guessing with extra steps. A team runs a test, sees the new version ahead after a few days, ships it, and records a win that never existed. Do that repeatedly and the site fills with changes that were noise, while the belief that testing works grows stronger.
It also protects you in the other direction. A result that fails to reach significance is not proof that the change did nothing; it usually means the test could not detect a difference of that size with the traffic available. Knowing which of those two situations you are in decides whether to run longer, test something bolder, or stop testing that page altogether.
Where statistical significance goes wrong
Peeking is the classic error. Checking daily and stopping the moment the tool reports a winner all but guarantees false positives, because with enough looks the threshold is crossed by chance sooner or later. Decide the run length in advance and leave the test alone.
Ending too early is the same mistake in another form. A test should cover whole weeks, so weekdays and weekends are represented, and it should avoid festivals, campaign launches and pricing changes that alter who arrives. Splitting the results afterwards to find a segment where the change won is another way of manufacturing a result that will not repeat.
The deepest misunderstanding is treating significance as importance. A confirmed but tiny improvement can be real and still not worth building. Significance says the difference exists; it does not say the difference pays.
How to act on it
Before starting an A/B test, write down what you are measuring, how long it will run, and the smallest improvement that would justify making the change permanent. Use a sample size calculator to check the test can detect a difference that size before you build anything.
Test differences big enough to matter — a different offer, a rewritten page, a shorter form — rather than shades of a button. If your traffic is modest, accept that formal testing will resolve slowly and lean on usability testing and recordings for evidence instead. Deciding which method a page deserves is part of ordinary conversion rate optimisation planning, and it is a better use of time than a test that can never conclude.
