Funnel & CRO
Check whether your A/B test winner is real or just random noise.
- Category
- Funnel & CRO
- Signup
- None
- Your data
- Stays in your browser
- Also in
- Claude, via MCP
How do you know if an A/B test result is statistically significant?
Run a two-proportion z-test on each variant's visitors and conversions. If the two-sided p-value is below 0.05, which this calculator shows as 95% confidence or higher, the difference is unlikely to be random noise. The result only holds if sample size was fixed in advance and you did not stop the test the moment it looked good.
Significant at 98.8% confidence (≥95%). Variant B's 25.4% uplift is very unlikely to be noise, safe to ship.
Two-proportion z-test for a single comparison. It doesn’t correct for peeking or multiple variants, decide your sample size before you start, not after you like the number.
About this calculator
A test that shows Variant B winning by 20% can still be pure noise if the sample is small, calling a winner too early is one of the most common and expensive CRO mistakes. This calculator runs a proper two-proportion z-test on your visitor and conversion counts and returns a confidence level, so "ship it" is backed by statistics instead of an eyeballed percentage difference.
How to use it
- Enter visitors and conversions for Variant A, your control.
- Enter visitors and conversions for Variant B, the test variant.
- Read the conversion rate for each variant, the relative uplift of B over A, and the statistical confidence that the difference is real.
- Only treat a result as significant at 95% confidence or higher, below that, keep the test running rather than acting on the number.
Methodology
Conversion rate for each variant is conversions ÷ visitors. Relative uplift is (rate B − rate A) ÷ rate A, expressed as a percentage change.
The significance test is a standard two-proportion z-test: a pooled conversion rate is computed across both variants, a standard error is derived from that pooled rate and each variant's sample size, and a z-score is calculated from the difference in rates divided by that standard error.
Confidence is derived from the z-score using a normal-distribution approximation (Abramowitz & Stegun), converting the z-score into the probability that the observed difference isn't due to chance.
This calculator runs a single, one-time comparison. It does not correct for "peeking", checking results repeatedly during a live test and stopping as soon as you see significance inflates false-positive rates well above the nominal threshold. Decide your sample size or test duration in advance and check once, not continuously.
It also only compares two variants. Testing three or more variants simultaneously requires a multiple-comparisons correction this tool doesn't apply, running several two-variant comparisons from one multi-variant test will overstate significance.
Two-proportion z-test used by the calculator
Rate A = Conversions A ÷ Visitors A; Rate B = Conversions B ÷ Visitors B Relative uplift = (Rate B − Rate A) ÷ Rate A Pooled rate p = (Conversions A + Conversions B) ÷ (Visitors A + Visitors B) Standard error SE = √( p × (1 − p) × (1 ÷ Visitors A + 1 ÷ Visitors B) ) z = (Rate B − Rate A) ÷ SE Confidence = (1 − two-sided p-value) × 100
- Visitors A, Visitors B
- Users (or sessions, consistently) assigned to the control and the test variant.
- Conversions A, Conversions B
- Users in each group who completed the goal. Count each user once.
- p
- The pooled conversion rate, the best estimate of the common rate if the variants were identical.
- z
- How many standard errors apart the two rates are. |z| above 1.96 corresponds to 95% confidence, two-sided.
This is the standard large-sample test for comparing two proportions described in the NIST/SEMATECH e-Handbook of Statistical Methods. It is two-sided, so it asks whether B is different from A, not only whether it is better. "Confidence" here is 1 minus the p-value: strictly, it is not the probability that B is truly better, but how surprising the observed gap would be if the variants performed the same. The normal approximation needs a reasonable number of conversions in each group; with only a handful of conversions per variant, treat the output as indicative only.
Worked example: a checkout button test
An online store tests a new checkout button. Variant A (control) gets 12,000 visitors and 360 orders. Variant B gets 12,000 visitors and 420 orders.
- Rate A = 360 ÷ 12,000 = 3.00%; Rate B = 420 ÷ 12,000 = 3.50%
- Relative uplift = (3.50% − 3.00%) ÷ 3.00% = 16.7%
- Pooled rate = 780 ÷ 24,000 = 3.25%
- SE = √(0.0325 × 0.9675 × (1 ÷ 12,000 + 1 ÷ 12,000)) = 0.00229
- z = 0.005 ÷ 0.00229 = 2.18
- Confidence = 97.1%, above the 95% threshold
The 16.7% uplift is statistically significant at 97.1% confidence. One caution: using Evan Miller's rule of thumb n = 16σ²/δ², detecting a 0.5-point lift on a 3% base needs about 18,600 visitors per variant (16 × 0.03 × 0.97 ÷ 0.005²). This test reached significance with fewer, which is possible when the true effect is large, but it also means the measured uplift is likely to be an overestimate. Expect the lift after rollout to be smaller than 16.7%.
How peeking changes the threshold you need
| Number of interim looks | Nominal significance needed for a true 5% false-positive rate |
|---|---|
| 1 | 2.9% |
| 2 | 2.2% |
| 3 | 1.8% |
| 5 | 1.4% |
| 10 | 1.0% |
What confidence level and sample size are good enough?
Most teams use 95% confidence for changes that affect revenue, and some testing platforms default lower: Optimizely says experiments are usually run at 90% significance and its sample size calculator uses that by default. Lowering the bar shortens tests but raises the false-positive risk, which Optimizely suggests only where a wrong call has little business impact.
Sample size matters as much as the threshold. Decide before launch the smallest lift worth detecting, calculate visitors per variant from it, and run for at least one full business cycle; Optimizely recommends a minimum of seven days so weekday and weekend behaviour are both included.
A decision rule: ship when confidence is at or above your pre-set threshold and the planned sample size was reached. If confidence is high but the sample is short, extend the test. If the planned sample is reached and confidence is low, treat it as no detectable difference and move on. For tests on paid traffic, the creative testing duration forecaster helps size the run.
Running tests in India vs the US
Indian sites often see sharp traffic and conversion swings around sale events such as Diwali and end-of-season sales, plus a large share of mobile and app traffic. Avoid starting or ending a test across a sale, because shoppers then behave differently from normal weeks and the winner may not hold. Payment mix also matters: if one variant changes how often shoppers choose cash on delivery, compare delivered orders, not just placed orders. Lower-traffic B2B sites may never reach the sample needed for small lifts, so test bigger changes. See conversion optimisation for how to prioritise.
US e-commerce tests face the same issue around Black Friday, Cyber Monday and the holiday season, when traffic quality and intent shift. Higher traffic volumes make it easier to reach sample size quickly, which tempts teams to stop early; the peeking problem applies regardless of volume. For B2B SaaS, the goal metric is often a demo request rather than revenue, so check that the winning variant also produces qualified pipeline in the CRM.
Common A/B testing mistakes
- Stopping the test as soon as it crosses 95%. Repeated checking inflates false positives; fix the sample size first or use a sequential method built for continuous monitoring.
- Running several variants and comparing each to control at 95% without correction, which raises the chance that at least one "wins" by luck.
- Counting sessions in one variant and users in the other, or counting one user's repeat conversions several times.
- Ignoring sample ratio mismatch. If a 50/50 split delivers 12,000 vs 10,500 visitors, the assignment may be broken and the result untrustworthy.
- Reading the observed uplift from a small test as the lift you will get after rollout. Small tests that reach significance usually overstate the effect.
- Testing changes too small for your traffic, then concluding nothing works when the test was never able to detect them. Use the conversion funnel calculator to find the step with the largest drop and test there first.
Sources
- 7.3.3. How can we determine whether two processes produce the same proportion of defectives?, NIST/SEMATECH e-Handbook of Statistical Methods. Accessed 2026-10-17.
- How Not To Run an A/B Test, Evan Miller. Accessed 2026-10-17.
- How long to run an experiment, Optimizely. Accessed 2026-10-17.
FAQ
95% is the standard threshold used across most CRO practice, corresponding to a 5% chance the observed difference is random noise. Some teams use 90% for lower-stakes changes and reserve 95%+ for anything touching pricing, checkout, or core conversion flows.
Checking a test's significance repeatedly and stopping the moment it crosses 95% dramatically inflates the real false-positive rate, sometimes to 30-40% instead of the intended 5%, because random fluctuation will cross any threshold at some point if you check often enough. Decide sample size or duration up front and evaluate once.
It's below the conventional 95% bar, meaning there's still a meaningful chance the result is noise. The calculator flags this as "trending," worth continuing to collect data on rather than shipping or killing based on that number alone.
No, this calculator uses both sample size and the observed conversion counts together, a huge sample with a tiny rate difference can still be non-significant, and a smaller sample with a large rate difference can already be significant. Always check the confidence output, not visitor count alone.
The p-value is the probability of seeing a difference at least as large as yours if the two variants actually perform the same. A p-value of 0.03 means a gap this big would appear about 3% of the time by chance alone. This calculator reports 1 minus the two-sided p-value as confidence.
It depends on your baseline conversion rate and the smallest lift you want to detect. Evan Miller's rule of thumb is n = 16 × p(1 − p) ÷ δ² per variant, where δ is the absolute difference. At a 3% base rate and a 0.5-point lift, that is about 18,600 visitors per variant; halving the lift you want to detect roughly quadruples the sample.
Two-tailed is the safer default because it detects a variant that is worse as well as better. A one-tailed test reaches significance with less data but assumes you only care about one direction, which is rarely true when a change could hurt conversion.
Yes, as long as the metric is a yes or no outcome per person, such as clicked or did not click. For revenue per visitor or order value, which are continuous metrics, a different test is needed. For email, compare clicks or orders rather than opens, which privacy features make unreliable.
Numbers looking off?
If the model says one thing and the pipeline says another, that gap is usually where the revenue is. A 30-minute call finds it.
It starts with a 30-minute call. Pick a time below.