A/B Test Statistical Significance Calculator
Check whether your A/B test winner is real or just random noise.
Significant at 98.8% confidence (≥95%). Variant B's 25.4% uplift is very unlikely to be noise, safe to ship.
Two-proportion z-test for a single comparison. It doesn't correct for peeking or multiple variants, decide your sample size before you start, not after you like the number.
About this calculator
A test that shows Variant B winning by 20% can still be pure noise if the sample is small, calling a winner too early is one of the most common and expensive CRO mistakes. This calculator runs a proper two-proportion z-test on your visitor and conversion counts and returns a confidence level, so "ship it" is backed by statistics instead of an eyeballed percentage difference.
How to use it
- Enter visitors and conversions for Variant A, your control.
- Enter visitors and conversions for Variant B, the test variant.
- Read the conversion rate for each variant, the relative uplift of B over A, and the statistical confidence that the difference is real.
- Only treat a result as significant at 95% confidence or higher, below that, keep the test running rather than acting on the number.
Methodology
Conversion rate for each variant is conversions ÷ visitors. Relative uplift is (rate B − rate A) ÷ rate A, expressed as a percentage change.
The significance test is a standard two-proportion z-test: a pooled conversion rate is computed across both variants, a standard error is derived from that pooled rate and each variant's sample size, and a z-score is calculated from the difference in rates divided by that standard error.
Confidence is derived from the z-score using a normal-distribution approximation (Abramowitz & Stegun), converting the z-score into the probability that the observed difference isn't due to chance.
This calculator runs a single, one-time comparison. It does not correct for "peeking", checking results repeatedly during a live test and stopping as soon as you see significance inflates false-positive rates well above the nominal threshold. Decide your sample size or test duration in advance and check once, not continuously.
It also only compares two variants. Testing three or more variants simultaneously requires a multiple-comparisons correction this tool doesn't apply, running several two-variant comparisons from one multi-variant test will overstate significance.
FAQ
95% is the standard threshold used across most CRO practice, corresponding to a 5% chance the observed difference is random noise. Some teams use 90% for lower-stakes changes and reserve 95%+ for anything touching pricing, checkout, or core conversion flows.
Checking a test's significance repeatedly and stopping the moment it crosses 95% dramatically inflates the real false-positive rate, sometimes to 30-40% instead of the intended 5%, because random fluctuation will cross any threshold at some point if you check often enough. Decide sample size or duration up front and evaluate once.
It's below the conventional 95% bar, meaning there's still a meaningful chance the result is noise. The calculator flags this as "trending," worth continuing to collect data on rather than shipping or killing based on that number alone.
No, this calculator uses both sample size and the observed conversion counts together, a huge sample with a tiny rate difference can still be non-significant, and a smaller sample with a large rate difference can already be significant. Always check the confidence output, not visitor count alone.