spyc.ai

Free tool

A/B Test Significance Calculator

Run a two-sided z-test on A and B to get the p-value, relative uplift and a confidence interval.

How it works

What is the A/B Test Significance Calculator?

The A/B Test Significance Calculator is a free tool that tells you whether the difference between two variants is likely to be real. Enter visitors and conversions for A and B. It runs a two-sided two-proportion z-test and returns both conversion rates, the relative uplift, the p-value and a confidence interval.

How to use the A/B Test Significance Calculator

  1. Enter visitors and conversions for A, the control.
  2. Enter visitors and conversions for B, the variant.
  3. Keep the 95% confidence level or change it.
  4. Read the p-value, uplift and confidence interval, then copy the results.

How do I know if an A/B test is statistically significant?

Compare the p-value with your significance threshold. At a 95% confidence level the threshold is 0.05: a p-value below it counts as significant, a p-value above it does not. The calculator makes that comparison for you and states the result in a sentence.

With 500 conversions from 10,000 visitors on A and 570 from 10,000 on B, the p-value is 0.0278, which is significant at 95% and not at 99%.

What does the p-value actually mean?

The p-value is the probability of seeing a difference at least this large if A and B truly converted at the same rate. A small p-value means the data would be surprising under that assumption.

It is not the probability that B is better, and it is not the probability that the result is a fluke. A p-value of 0.03 does not mean a 97% chance that the variant wins.

How does this calculator test significance?

It uses a two-proportion z-test. The two conversion rates are compared using a pooled standard error, the z-score is converted to a two-sided p-value with the normal distribution, and the confidence interval for the difference uses the unpooled standard error.

The normal approximation needs a reasonable number of conversions and non-conversions in each group. With only a handful in either, the result is unreliable and an exact test is the better choice.

What does the confidence interval tell me?

It gives a range of plausible values for the true difference between B and A, in percentage points. In the example above the 95% interval runs from about +0.08 to +1.32 points: probably a real gain, possibly a very small one.

The interval is often more useful than the p-value. A significant result whose interval starts just above zero is a much weaker basis for a decision than the headline uplift suggests.

Why is my A/B test not significant?

Usually because the sample is too small for the size of the effect. A result that is not significant does not show the variants are equal; it shows the test could not tell them apart.

Work out the visitors you need before the next test with the A/B Test Sample Size Calculator. And do not keep checking a running test and stop the moment it turns significant, because that inflates false positives. If the test is an ad holdout, the Incrementality Test Calculator reports lift and cost per incremental conversion.

Questions

Is the test one-sided or two-sided?
Two-sided. It asks whether B differs from A in either direction, which is the standard, more conservative choice.
What confidence level should I use?
95% is the common convention. Use 99% when a wrong call would be costly, or 90% for low-risk decisions, and choose before looking at the data.
Can I use it for revenue per visitor?
No. It compares conversion rates, where each visitor either converts or does not. Averages such as revenue per visitor need a different test.
Do A and B need the same number of visitors?
No. The test handles unequal groups. A large unplanned imbalance is worth investigating, though, because it can mean the split is broken.
Is the A/B Test Significance Calculator free?
Yes, with a free spyc.ai account. It runs in your browser, so nothing you type is sent to a server.

Marketing Calculators

More in Marketing Calculators

ROAS Calculator

Work out return on ad spend from revenue and spend, plus break-even ROAS and profit from your gross margin.

Calculate ROAS

CAC and LTV Calculator

Get CAC, lifetime value, the LTV:CAC ratio and payback from spend, new customers, ARPU, margin and churn.

Calculate LTV:CAC

Churn Rate Calculator

Measure customer and revenue churn for a month, quarter or year, with compounded monthly and annual rates.

Calculate churn

Ready when you are

Before your next buyer asks AI about you, ask AI about yourself

One check, four engines, about twenty seconds. We show you the answers before we ask for anything.