Analytics and Tracking

p-value

Also called probability value, significance level

The probability of seeing a result at least this extreme if the change you tested had made no difference.

Quick facts: p-value

Category
Analytics and Tracking
Also called
probability value, significance level
Level
Advanced
Affects
Test decisions, confidence in results, wasted development work
Where to see it
Testing platforms such as VWO or Optimizely, free significance calculators, spreadsheets
In this article4
  1. What a p-value measures
  2. Why the p-value matters
  3. Where the p-value is misunderstood
  4. How to use it properly

What a p-value measures

Every test starts from a deliberately sceptical position, called the null hypothesis: the change made no difference, and any gap between the two versions is just the luck of who arrived. The p-value asks how surprising your actual result would be if that sceptical position were true.

A small p-value means the observed gap would rarely appear by chance alone, so the sceptical explanation starts to look strained. A large p-value means a gap that size turns up easily without any real effect, so the result tells you nothing. The value falls as the difference between versions grows and as the amount of traffic behind it grows, which is why a big effect on modest traffic and a small effect on heavy traffic can produce a similar figure.

Testing tools rarely print the p-value itself. They convert it into a confidence or probability-to-beat display, but the underlying calculation is the same.

Why the p-value matters

It gives a consistent rule for deciding when a difference is worth acting on, replacing the judgement of whoever is looking at the dashboard. Agreed in advance, it stops a test being called for the version the team already preferred.

It also communicates honestly to a client or a board. Saying that a result of this size would be unusual if the change did nothing is a claim you can defend, unlike saying the new page simply looks better.

Where the p-value is misunderstood

It is not the probability that your change works, and it is not the probability that you are right. It only describes how likely data like yours would be under the assumption that nothing happened. That distinction sounds academic until someone reports it the wrong way round to a client, and it is the most common error in the whole subject.

Nor does it say anything about size or value. A test can clear the threshold on a difference too small to be worth the development work, and a genuinely large improvement can miss the threshold entirely because the traffic was thin. The other trap is looking repeatedly and stopping when the figure dips below the line. Checked often enough it will dip eventually, whether or not anything real is happening.

How to use it properly

Fix the threshold, the primary measure and the required sample size before the test starts, then leave the result alone until the sample is reached. Judge one measure by this rule, not every metric on the report, because testing many measures makes a chance result somewhere almost certain.

When the threshold is met, ask the second question the p-value cannot answer: is the size of the improvement worth the cost of shipping it? Report the difference and its range alongside the verdict, and treat the whole thing as one input to the decision rather than the decision itself. Used this way it keeps a conversion rate optimisation programme honest.

Do and do not

Do

  • Agree the threshold before the test begins
  • Apply it to one primary measure only
  • Report the size of the difference alongside the verdict

Do not

  • Call it the probability that your change works
  • Move the threshold once you have seen the result
  • Check daily and stop the moment it dips

Questions people ask about this

Does a low p-value mean my change definitely works?

No. It means a difference of that size would be unusual if the change had done nothing, which is good evidence but not proof. Chance results still occur, particularly when many measures are examined or a test is checked repeatedly. Treat it as strong support for a decision, not a guarantee.

What threshold should I use?

Pick one before the test starts and apply it to a single primary measure. A stricter threshold means fewer false winners but demands more traffic; a looser one finds more, including some that are not real. What matters most is that the line is agreed in advance and not moved once the result appears.

My test never reaches significance. What should I do?

Usually the change was too subtle for your traffic. Test something bolder, such as the offer, the pricing presentation or the form length, rather than a wording tweak. You can also measure an action that happens more often, like reaching the enquiry step, so the test has more events to work with.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.