Free tool
A/B Test Sample Size Calculator
Find the visitors each variant needs from baseline rate, minimum detectable effect, significance and power.
How it works
What is the A/B Test Sample Size Calculator?
The A/B Test Sample Size Calculator is a free tool that tells you how many visitors each variant needs before a test can reliably detect a given change. Enter the baseline conversion rate, the minimum detectable effect, the significance level and the power. It uses a two-sided two-proportion z-test.
How to use the A/B Test Sample Size Calculator
- Enter your current conversion rate as the baseline.
- Enter the smallest effect worth detecting and choose relative or absolute.
- Keep 95% confidence and 80% power, or change them.
- Optionally add daily visitors to see an estimated run time, then copy the results.
How do I calculate sample size for an A/B test?
You need four inputs: the baseline conversion rate, the smallest change you care about, the significance level and the statistical power. The calculator applies the standard formula for comparing two proportions with a two-sided z-test:
n per variant = (z for significance x sqrt(2 x p x (1 - p)) + z for power x sqrt(p1 x (1 - p1) + p2 x (1 - p2)))^2 / (p2 - p1)^2, where p1 is the baseline, p2 the variant rate and p their average. Going from 10% to 12% at 95% confidence and 80% power needs 3,841 visitors per variant.
What is minimum detectable effect?
Minimum detectable effect, MDE, is the smallest difference the test is designed to find. It can be relative or absolute, and confusing the two is a classic mistake. On a 5% baseline, a 10% relative effect means 5.5%, while a 10 point absolute effect means 15%.
The calculator has a switch for this and shows the variant rate it implies, so you can check you asked for what you meant.
What do significance level and statistical power mean?
The significance level sets the false positive rate. At 95% confidence, a test of two identical pages will still report a significant difference about 5% of the time. Power is the chance of detecting an effect that really exists and is at least as large as the MDE; at 80% power you will miss such an effect about one time in five.
95% and 80% are conventions, not laws. Raising either one raises the sample size.
Why is the required sample size so large?
Because sample size grows with the inverse square of the effect. Halving the minimum detectable effect roughly quadruples the visitors you need. Low baseline rates make it worse, since conversions are rare events and it takes many visitors to see enough of them.
If the number is out of reach, the realistic options are to test a bolder change, test on a higher-traffic page, or measure an earlier step that converts more often.
How long should I run an A/B test?
Until every variant has reached the planned sample size, and for whole weeks, so weekday and weekend behaviour are both covered. Add your daily visitors and the calculator estimates the run time for two variants with an even split.
Do not stop early because the result looks significant. Checking repeatedly and stopping at the first significant reading inflates the false positive rate well beyond the level you chose. When the test is done, analyse it with the A/B Test Significance Calculator. For ad holdout tests, see the Incrementality Test Calculator.
Questions
- Is the sample size per variant or in total?
- Per variant. The calculator also shows the total for a two-variant test, which is double.
- Does the calculator use a one-sided or two-sided test?
- Two-sided. It plans for detecting a change in either direction, which is the safer default and needs more visitors than a one-sided test.
- What if I am testing more than two variants?
- Each variant still needs at least the per-variant size. Comparing several variants with the control also raises the false positive risk, so consider a stricter significance level.
- Why do other calculators give a slightly different number?
- There are several accepted formulas: pooled or unpooled variance, with or without a continuity correction, one-sided or two-sided. Results usually differ by a few percent.
- Is the A/B Test Sample Size Calculator free?
- Yes, with a free spyc.ai account. It runs in your browser, so nothing you type is sent to a server.
Marketing Calculators
More in Marketing Calculators
ROAS Calculator
Work out return on ad spend from revenue and spend, plus break-even ROAS and profit from your gross margin.
Calculate ROASCAC and LTV Calculator
Get CAC, lifetime value, the LTV:CAC ratio and payback from spend, new customers, ARPU, margin and churn.
Calculate LTV:CACChurn Rate Calculator
Measure customer and revenue churn for a month, quarter or year, with compounded monthly and annual rates.
Calculate churnReady when you are
Before your next buyer asks AI about you, ask AI about yourself
One check, four engines, about twenty seconds. We show you the answers before we ask for anything.