Results
- Conversion rate A
- –
- Conversion rate B
- –
- Relative change
- –
- p-value
- –
- Confidence interval
- –
What significance actually tells you
The p-value answers one narrow question: if the two variants were truly identical, how often would you see a difference at least this large by chance alone? A p-value below your threshold means chance is an unconvincing explanation. It does not tell you the size of the improvement, how likely it is to hold, or whether it is worth shipping.
The confidence interval is the more useful number and the one most reports leave out. It is the range of true differences consistent with your data. An interval running from a small decline to a large gain is not a win, however green the significance badge is, because a real decline is still one of the outcomes your data supports.
Peeking is what breaks most tests
Checking the result every day and stopping the moment it crosses the threshold makes false positives far more likely, because a test that fluctuates will eventually cross the line by chance even when nothing is happening. The mathematics behind the p-value assumes one look, at the end, on a sample size you fixed beforehand.
So decide two things before the test starts: how long it runs, and how many conversions each variant needs. Then let it finish. Watching it is fine; stopping it early on what you see is not, unless something is genuinely broken.
Test size, and what is worth testing
The smaller the change you are hoping to detect, the more traffic you need to see it, and the relationship is unforgiving. Most sites do not have the traffic to resolve a small difference in any reasonable period, which is an argument for testing bigger changes rather than for testing more often. A different offer, a different page structure or a different form length moves numbers far enough to be measurable; a button colour usually does not.
When the traffic is not there at all, the honest answer is that testing is the wrong instrument, and the improvements should come from research, session recordings and fixing what is obviously broken. Deciding which of those a site needs is the first step of the conversion rate optimisation service, and it depends on the tracking being right, which is what analytics and tracking covers.
