Question 1

What is statistical significance in A/B testing?

Accepted Answer

Statistical significance means the difference between your variants is unlikely to be caused by chance. At 95% confidence, there’s only a 5% probability the result is noise. This calculator uses a two-proportion z-test, which is the standard method for comparing two conversion rates.

Question 2

How many visitors do I need for an A/B test?

Accepted Answer

It depends on your baseline conversion rate and how small a change you want to detect. A page converting at 3% needs roughly 13,000 visitors per variant to detect a 20% relative improvement. Lower conversion rates and smaller effects need more traffic.

Question 3

What’s the difference between frequentist and Bayesian A/B test results?

Accepted Answer

The frequentist result (confidence level, p-value) tells you whether to reject the hypothesis that A and B are the same. The Bayesian result tells you the probability that B is better than A. The Bayesian number is usually easier to act on: "there’s a 96% chance B wins" is more useful than "p = 0.038."

Question 4

How long should I run my A/B test?

Accepted Answer

Until you hit your required sample size, or at minimum one full business cycle (usually 1–2 weeks). Ending a test early because it looks like a winner is the most common A/B testing mistake. Early results are unreliable.

Question 5

What’s a good minimum detectable effect for A/B testing?

Accepted Answer

Set it to the smallest improvement that would actually matter to your business. For most landing pages, 10–20% relative lift is a reasonable target. Going smaller means you need significantly more traffic.

Free A/B Test Calculator

How to use this calculator

Enter your data

Read the verdict

Plan your next test

What this A/B test calculator measures

Statistical significance (frequentist)

Probability of winning (Bayesian)

Sample size and test duration

Frequently asked questions

Get the A/B Testing Playbook

Run smarter A/B tests with Leadpages