What a p-value measures
Every test starts from a deliberately sceptical position, called the null hypothesis: the change made no difference, and any gap between the two versions is just the luck of who arrived. The p-value asks how surprising your actual result would be if that sceptical position were true.
A small p-value means the observed gap would rarely appear by chance alone, so the sceptical explanation starts to look strained. A large p-value means a gap that size turns up easily without any real effect, so the result tells you nothing. The value falls as the difference between versions grows and as the amount of traffic behind it grows, which is why a big effect on modest traffic and a small effect on heavy traffic can produce a similar figure.
Testing tools rarely print the p-value itself. They convert it into a confidence or probability-to-beat display, but the underlying calculation is the same.
Why the p-value matters
It gives a consistent rule for deciding when a difference is worth acting on, replacing the judgement of whoever is looking at the dashboard. Agreed in advance, it stops a test being called for the version the team already preferred.
It also communicates honestly to a client or a board. Saying that a result of this size would be unusual if the change did nothing is a claim you can defend, unlike saying the new page simply looks better.
Where the p-value is misunderstood
It is not the probability that your change works, and it is not the probability that you are right. It only describes how likely data like yours would be under the assumption that nothing happened. That distinction sounds academic until someone reports it the wrong way round to a client, and it is the most common error in the whole subject.
Nor does it say anything about size or value. A test can clear the threshold on a difference too small to be worth the development work, and a genuinely large improvement can miss the threshold entirely because the traffic was thin. The other trap is looking repeatedly and stopping when the figure dips below the line. Checked often enough it will dip eventually, whether or not anything real is happening.
How to use it properly
Fix the threshold, the primary measure and the required sample size before the test starts, then leave the result alone until the sample is reached. Judge one measure by this rule, not every metric on the report, because testing many measures makes a chance result somewhere almost certain.
When the threshold is met, ask the second question the p-value cannot answer: is the size of the improvement worth the cost of shipping it? Report the difference and its range alongside the verdict, and treat the whole thing as one input to the decision rather than the decision itself. Used this way it keeps a conversion rate optimisation programme honest.
