GoldenrozBook a call

A/B test significance calculator

Enter visitors and conversions for variant A and variant B. The calculator runs a two-sided, two-proportion z-test and tells you whether the difference clears the 95% level or could be chance.

The starting values are examples, not benchmarks. Replace them with your own figures. Nothing you enter is stored or sent anywhere.

  • Conversion rate A
    Conversions divided by visitors
  • Conversion rate B
    Conversions divided by visitors
  • Relative lift of B over A
    Difference in rates divided by rate A
  • Z-score
    Difference in rates divided by its pooled standard error
  • P-value
    Chance of a gap this large if the variants were equal
  • Confidence
    1 minus the p-value

How to read the result

The example values show the usual trap. Variant B converts 15% better than A, which looks like a clear win, yet the p-value is above 0.05. With 300 and 345 conversions, a gap that size appears by chance about one time in fourteen. Calling the test now means acting on noise.

Decide the sample size before the test starts and check significance once, at the end. Looking every day and stopping the first time the result crosses 95% produces false winners, because random swings cross that line often in the first days.

Significance is not the same as importance. A significant 2% lift can be too small to matter, and the test measures the conversion you counted, not profit. For ad creative, the practical routine is in the creative testing framework. To test whether a channel adds sales at all, see incrementality testing.

Questions about A/B test significance

What does statistically significant mean in an A/B test?

It means the difference between the variants is larger than chance alone would plausibly produce. At the 95% level, a gap that big would show up less than 5% of the time if the two variants truly performed the same.

What is a p-value?

The probability of seeing a difference at least as large as the one you measured if there were no real difference between the variants. A p-value of 0.03 means a 3% chance. Below 0.05 is the usual threshold for calling a result significant.

How many conversions do I need?

It depends on the size of the lift you want to detect. Small lifts need large samples: detecting a 10% relative lift on a 3% conversion rate takes tens of thousands of visitors per variant. A few hundred conversions per variant is a practical minimum for most tests.

Which test does this calculator use?

A two-sided z-test for two proportions with a pooled standard error. It suits tests with one conversion per visitor at most and at least a few dozen conversions per variant. It is not meant for revenue per visitor or for tests with more than two variants.

Related calculators

  • Cost per acquisition from ad spend, cost per click and conversion rate, with the CPC or conversion rate needed to hit a target.
  • ROAS, break-even ROAS, profit after ad spend and cost per order from spend, revenue and margin.
  • The ad spend, clicks and orders a revenue goal requires, checked against break-even ROAS.

All free calculators

Talk to the person who will do the work

One hour, no sales team. You talk to Daniel Rozin, we look at your ad accounts and tracking, and you leave with a clear next step.

Or send the details

Book a callWhatsApp us