Saved

Press the star next to a tool's name to save it.

All my tools

A/B Test Significance Calculator

Enter the visitors and conversions for each version, and this A/B test significance calculator shows the conversion rates, the uplift, the p-value and whether the result is significant.

Restored your last values ·

A is your original version and B is the variation.

Result

–

Enter your test numbers.

See full result

An A/B test significance calculator tells you whether the winner of your test really won, or whether the gap could be down to luck. Enter the visitors and conversions for each version, and this A/B test significance calculator shows the conversion rates, the uplift, the z score and the p-value, along with a confidence interval for the difference. Marketers use it to judge landing pages, email subject lines and ads, and product teams use it to evaluate new features. All of the math runs in your browser, so your test numbers stay private.

How to use the A/B test significance calculator

  1. Enter the visitors and conversions for version A, your original.
  2. Enter the same numbers for version B, your variation.
  3. Choose a confidence level. 95% is the most common.
  4. Press Calculate and read the result.

What is A/B testing?

A/B testing, also known as split testing, is a randomized experiment that usually involves two variants. Visitors are randomly shown either A or B, and you then compare how each group behaves. A/B tests are widely considered the simplest form of controlled experiment. Random assignment is what makes the comparison fair, because the two groups should differ only in the version that they saw.

The formula

  • Conversion rate = conversions ÷ visitors
  • Pooled rate = total conversions ÷ total visitors
  • Standard error = √(pooled × (1 − pooled) × (1 ÷ visitors A + 1 ÷ visitors B))
  • Z score = (rate B − rate A) ÷ standard error
  • P-value = the two-sided probability from the normal distribution

A worked example

Suppose that version A had 1,000 visitors and 100 conversions, a rate of 10%. Version B also had 1,000 visitors but got 130 conversions, a rate of 13%. That's a difference of 3 percentage points and a relative uplift of 30%. The pooled rate is 11.5%, the standard error is about 0.0143, and the z score is about 2.10. That gives a p-value of about 0.035. Since it's below 0.05, the result is statistically significant at the 95% level. The 95% interval for the difference runs from about +0.21 to +5.79 percentage points.

What does the p-value mean?

The p-value is the probability of obtaining results at least as extreme as the ones you observed, assuming that there's actually no difference between the versions. A very small p-value means that such an outcome would be very unlikely if the two versions truly performed the same. A threshold, often called alpha, is set before examining the data, and it's commonly 0.05. The p-value isn't the probability that B is better, and it says nothing about how large or valuable the improvement is.

Reading the confidence interval

The interval shows the range of differences that are consistent with your data. If it includes zero, the result isn't significant at that level. A wide interval means that your estimate is imprecise, usually because the sample is small. Even when a test is significant, look at the low end of the interval and ask whether an improvement of that size would still be worth the cost of making the change.

Choosing a confidence level

ConfidenceP-value neededTypical use
90%Below 0.10Low-risk changes, faster decisions
95%Below 0.05The usual standard
99%Below 0.01Costly or hard-to-reverse changes

Good testing habits

  • Plan ahead. Decide on your sample size, your test length and your main metric before you start.
  • Don't stop early. Checking repeatedly and stopping at the first significant result raises the rate of false wins.
  • Run for full weeks. Behavior often differs between weekdays and weekends.
  • Change one thing at a time where possible, so that you know what caused the difference.
  • Keep the split random and stable, so that each person always sees the same version.
  • Check for tracking errors if your two groups are very different in size.

Why small tests mislead

With only a few conversions, the rates can swing wildly from pure chance. Going from 2 conversions to 3 is a 50% lift on paper, but it means almost nothing. As your sample grows, the standard error shrinks, and it becomes possible to detect smaller differences. Very large samples have the opposite problem, as even a trivial difference can reach significance. That's why it's always worth looking at the size of the effect alongside the p-value.

Limits of this calculator

The test assumes that visitors are independent and randomly assigned, that each one either converts or doesn't, and that the sample is reasonably large. For very small samples, Fisher's exact test is a better choice. If you're comparing several variations at once, the chance of a false positive rises, and a correction is needed. Revenue per visitor and other continuous metrics need a different test.

Common mistakes

  • Calling a winner too soon. Early results often fade.
  • Confusing significance with importance. A tiny gain may not be worth shipping.
  • Testing many ideas without correcting for it. Some will “win” by chance.

Measure your data with the average calculator, find changes with the percentage calculator, and test title ideas with the title tag checker.

What next

Use this often? Add A/B Test Sig to your Home screen.

Frequently asked questions

How does the A/B test significance calculator work?

It runs a two-proportion z-test on your visitors and conversions to find a p-value.

What p-value is significant?

A value below 0.05 is the most common threshold, which matches 95% confidence.

How many visitors do I need?

It depends on your conversion rate and the size of the effect, but small tests are unreliable.

Can I stop my test as soon as it is significant?

It's best not to, as this raises the risk of a false win.

Is my data uploaded?

No. The calculation runs in your browser.

What went wrong?

We only receive the reason you pick, never what you typed into the tool.

Thanks for your feedback.

More Math Tools