Free tool
Incrementality Test Calculator
Turn test and holdout results into incremental lift, incremental conversions, cost per result and iROAS.
How it works
What is the Incrementality Test Calculator?
The Incrementality Test Calculator is a free tool that measures what your ads actually caused. Enter the size and conversions of a test group that saw ads and a holdout group that did not. It returns incremental lift, incremental conversions, a p-value and, with spend and conversion value, cost per incremental conversion and iROAS.
How to use the Incrementality Test Calculator
- Enter the size of the test group that saw ads and its conversions.
- Enter the size of the holdout group that saw no ads and its conversions.
- Add the ad spend on the test group and the revenue per conversion.
- Read lift, incremental conversions, cost per incremental conversion and iROAS.
What is incrementality testing?
Incrementality testing measures the conversions that happened because of your advertising, as opposed to conversions that would have happened anyway. You randomly withhold ads from a holdout group, show them to a test group, and compare conversion rates.
The holdout group's rate is the baseline: what people do with no ads. Anything the test group does above that is incremental.
How do I calculate incremental lift?
Incremental lift = (test conversion rate - holdout conversion rate) / holdout conversion rate. Incremental conversions = (test rate - holdout rate) x test group size.
With 2,160 conversions from 90,000 people in the test group and 200 from 10,000 in the holdout, the rates are 2.4% and 2.0%. Lift is 20% and the ads caused 360 conversions. The other 1,800 would have happened without them. Because the tool compares rates, the two groups do not need to be the same size.
What is iROAS?
iROAS is incremental return on ad spend: incremental revenue divided by ad spend. It counts only the revenue the ads caused. In the example, 360 incremental conversions worth 100 each on 18,000 of spend gives an iROAS of 2x and a cost per incremental conversion of 50.
Platform-reported ROAS for the same campaign would count all 2,160 conversions it could claim, and so would look far better. Compare the two with the ROAS Calculator.
Is my incrementality result statistically significant?
The calculator runs a two-sided two-proportion z-test on the two groups and shows the p-value. Below 0.05 the difference is significant at the 95% level. Above it, the measured lift may be noise, however good the iROAS looks.
Small holdout groups are the usual reason for an inconclusive result. Plan the group sizes first with the A/B Test Sample Size Calculator.
What can go wrong with a holdout test?
The result is only valid if people were assigned to the two groups at random. Comparing people who happened to see ads with people who did not is not a holdout test, because ad platforms deliberately show ads to the people most likely to convert.
Other common problems are holdout users being reached through another channel, and tests that end before slower buyers have converted. The large ad platforms offer built-in lift studies that handle the random split for you; this calculator works on the numbers from any such test.
Questions
- How big should the holdout group be?
- Big enough to detect the lift you expect. A holdout of around 10% is a common starting point, but the right size depends on your conversion rate and the lift, so calculate it.
- Can incremental lift be negative?
- Yes. If the holdout converts at a higher rate than the test group, lift is negative. With small groups that is usually noise, not proof the ads did harm.
- What is the difference between ROAS and iROAS?
- ROAS divides all attributed revenue by spend. iROAS divides only the revenue the ads caused by spend, so it is lower and more honest.
- Do I need spend and revenue figures?
- No. Lift, incremental conversions and the p-value need only the group sizes and conversions. Spend adds cost per incremental conversion; conversion value adds iROAS.
- Is the Incrementality Test Calculator free?
- Yes, with a free spyc.ai account. It runs in your browser, so nothing you type is sent to a server.
Marketing Calculators
More in Marketing Calculators
ROAS Calculator
Work out return on ad spend from revenue and spend, plus break-even ROAS and profit from your gross margin.
Calculate ROASCAC and LTV Calculator
Get CAC, lifetime value, the LTV:CAC ratio and payback from spend, new customers, ARPU, margin and churn.
Calculate LTV:CACChurn Rate Calculator
Measure customer and revenue churn for a month, quarter or year, with compounded monthly and annual rates.
Calculate churnReady when you are
Before your next buyer asks AI about you, ask AI about yourself
One check, four engines, about twenty seconds. We show you the answers before we ask for anything.