A/B Test Significance Calculator

A free Excel calculator for checking statistical significance before scaling or killing campaign tests.

For anyone running landing-page, ad or email tests who needs a yes or no before shipping the winner.

How to use it

1. Open the Calculator tab. The How To Use tab explains the logic; the Calculator tab is where you work.
2. Fill the red cells. Enter visitors and conversions for Variation A and Variation B. Nothing else needs editing.
3. Read steps 2 to 4. The sheet shows each conversion rate and its standard error, the 90% and 95% ranges, then a plain YES or NO for each confidence level, with the Z score and p-value.

Worked example: The numbers the file ships with

A: 5,000 visitors, 802 conversions (16.04%). B: 5,001 visitors, 801 conversions (16.02%). The 95% ranges are 15.02–17.06% and 15.00–17.03%, almost the same. Z = 0.03, p-value = 0.51, so the sheet says NO at both 90% and 95%. There is no winner yet; keep the test running or stop treating B as better.

Questions

How many visitors do I need?

Enough that the two ranges stop overlapping. With conversion rates around 2–5%, that is usually thousands of visitors per variation, not hundreds. If the sheet still says NO after a full business cycle, the difference is probably too small to matter.

Should I use 90% or 95%?

Use 95% for changes that are expensive to undo (pricing, a new landing page). 90% is fine for low-risk changes like ad copy, where a wrong call costs little.

Can I stop the test as soon as it says YES?

Not on day one. Let it run at least a full week so weekday and weekend traffic are both in it, then check again.

Related: Simple Digital Media Plan · Google Ads Optimization Roadmap

View all marketing assets →