Conversion and UX

Statistical Significance

Also called Confidence level, significance

The confidence that a difference between two test versions is real rather than the result of chance.

Quick facts: Statistical Significance

Category
Conversion and UX
Also called
Confidence level, significance
Level
Advanced
Affects
Test decisions, wasted development, reported wins
Where to see it
Testing platform reports, sample size calculators
In this article4
  1. What statistical significance measures
  2. Why statistical significance matters
  3. Where statistical significance goes wrong
  4. How to act on it

What statistical significance measures

Run the same page against itself, showing identical content to two random halves of your traffic, and the two halves will still produce different conversion rates. People arrive in different moods, on different days, from different sources. Some difference is always there, and it means nothing.

Statistical significance is the calculation that separates that ordinary variation from a real effect. It asks how likely a gap this large would be if the two versions were genuinely identical. When that likelihood is small enough, the result is called significant, and the convention is to fix the threshold before the test starts rather than after seeing the numbers.

Three things drive it: how many people saw each version, how often they converted, and how large the gap is. A big difference shows up quickly. A small one may need more traffic than your site will ever produce in a sensible period.

Why statistical significance matters

Without it, testing quietly turns into guessing with extra steps. A team runs a test, sees the new version ahead after a few days, ships it, and records a win that never existed. Do that repeatedly and the site fills with changes that were noise, while the belief that testing works grows stronger.

It also protects you in the other direction. A result that fails to reach significance is not proof that the change did nothing; it usually means the test could not detect a difference of that size with the traffic available. Knowing which of those two situations you are in decides whether to run longer, test something bolder, or stop testing that page altogether.

Where statistical significance goes wrong

Peeking is the classic error. Checking daily and stopping the moment the tool reports a winner all but guarantees false positives, because with enough looks the threshold is crossed by chance sooner or later. Decide the run length in advance and leave the test alone.

Ending too early is the same mistake in another form. A test should cover whole weeks, so weekdays and weekends are represented, and it should avoid festivals, campaign launches and pricing changes that alter who arrives. Splitting the results afterwards to find a segment where the change won is another way of manufacturing a result that will not repeat.

The deepest misunderstanding is treating significance as importance. A confirmed but tiny improvement can be real and still not worth building. Significance says the difference exists; it does not say the difference pays.

How to act on it

Before starting an A/B test, write down what you are measuring, how long it will run, and the smallest improvement that would justify making the change permanent. Use a sample size calculator to check the test can detect a difference that size before you build anything.

Test differences big enough to matter — a different offer, a rewritten page, a shorter form — rather than shades of a button. If your traffic is modest, accept that formal testing will resolve slowly and lean on usability testing and recordings for evidence instead. Deciding which method a page deserves is part of ordinary conversion rate optimisation planning, and it is a better use of time than a test that can never conclude.

Do and do not

Do

  • Fix the run length before the test starts
  • Run over whole weeks, including weekends
  • Set the smallest improvement worth shipping in advance

Do not

  • Stop the test the day it looks ahead
  • Hunt for a winning segment after the event
  • Treat a significant result as an important one

Questions people ask about this

How long should I run a test before calling a winner?

Decide before you start, using a sample size calculator based on your current conversion rate and the smallest improvement worth having. Then run for whole weeks so weekday and weekend behaviour are both included, and avoid festivals or campaign changes during the period. Stopping when the result first looks good is how false winners get shipped.

My test is significant but the difference is tiny. Should I ship it?

Significance and value are separate questions. A real but very small gain may cost more to build and maintain than it returns, and it will be invisible against normal fluctuations in your reporting. Judge the change on whether the effort is justified by the outcome, then decide, rather than shipping because the tool said significant.

What if my site does not get enough traffic to reach significance?

Then formal testing is the wrong tool for most decisions on that site. Test only large changes that could produce a big difference, and gather evidence in other ways: watching people attempt the task, reading form abandonment data, and asking customers what nearly stopped them. Those methods do not need volume to be useful.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.