Is my test result statistically significant?. The A/B Test Significance Calculator takes visitors in a (control), conversions in a, visitors in b (variant), conversions in b and returns p-value (two-tailed) plus result, conversion rate a, conversion rate b, relative lift of b over a, z-score. Results update as you type, and the formula is shown under the result so you can repeat it in your own spreadsheet.
Marketing metrics are most useful as trends and comparisons between channels. Measure every campaign the same way, over the same period, and judge it against your gross margin rather than against revenue alone. Use the worked example below to check the maths against your own figures.
How the A/B Test Significance Calculator works
A two-proportion z-test tells you whether the difference between two conversion rates is bigger than chance would produce. Below a p-value of 0.05 the result is conventionally called significant; below 0.01 it is strong.
Worked example
With the example values (visitors in a (control) of 5000, conversions in a of 250, visitors in b (variant) of 5000, conversions in b of 290), the p-value (two-tailed) is 0.08; result Not significant at 95%: keep testing or accept no difference, conversion rate a 5%, conversion rate b 5.80%, relative lift of b over a 16%, z-score 1.77. Change any figure above and the result updates immediately; use Copy results to paste the summary into a note or email.
Assumptions and limits: Two-tailed z-test for proportions; assumes independent samples and enough conversions (30+) in each group.
Related Calculators
Frequently Asked Questions
How is p-value (two-tailed) calculated?
z = (pB − pA) ÷ √(p(1 − p)(1/nA + 1/nB)), with p the pooled rate; p-value from the normal distribution.
Which figures do I need?
Visitors in a (control), conversions in a, visitors in b (variant), conversions in b. Take them from the same period and the same set of accounts or reports so the ratio is consistent, and check the example values as a guide to the units expected.
How often should I track this metric?
Weekly for live campaigns, monthly for channel comparisons, and always over the same period for spend and results so the figures line up.
What assumptions does this calculator make?
Two-tailed z-test for proportions; assumes independent samples and enough conversions (30+) in each group.
