Central Limit Theorem and A/B Test Noise
Your product has a baseline sign-up rate of 8%. A colleague runs an experiment with 2,000 users in control and 2,000 in treatment and reports "treatment converted at 8.9% vs 8.0% — an 11% relative lift, ship it!"
- Using the CLT, compute the standard error of each arm's conversion rate and the standard error of the difference between arms.
- How many standard errors is the observed 0.9 percentage-point difference? What does that say about whether the lift is real?
- The colleague proposes fixing this by running the test on 4× as many users. By what factor does the standard error shrink? What if they wanted the standard error to be 10× smaller?
- Explain in one or two sentences why the CLT — not the distribution of individual sign-ups — is what justifies these calculations.
1. Standard errors via the CLT
Each arm's conversion rate is a mean of Bernoulli(p) draws, so by the CLT it is approximately Normal with \mathrm{SE} = \sqrt{p(1-p)/n}. Using the pooled/baseline p \approx 0.08:
\mathrm{SE}_{\text{arm}} = \sqrt{0.08 \times 0.92 / 2000} = \sqrt{0.0000368} \approx 0.0061 (0.61 percentage points).
The arms are independent, so variances add: \mathrm{SE}_{\text{diff}} = \sqrt{0.0061^2 + 0.0061^2} \approx 0.0086 (0.86 percentage points).
2. How many standard errors?
0.009 / 0.0086 \approx 1.05 standard errors. Under a Normal, a difference of about one SE happens frequently by chance alone (roughly 30% of the time in either direction). The observed lift is entirely consistent with no true difference. The relative-lift framing ("11%") hides that the absolute effect is inside the noise floor. Whether to call this "not significant", and what threshold to use, is the business of the Hypothesis Testing and A/B Testing subjects — but the arithmetic here already tells you the data cannot distinguish 8.9% from 8.0%.
3. Scaling the sample
SE scales with 1/\sqrt{n}. Four times the users → SE shrinks by \sqrt{4} = 2, to about 0.43 pp for the difference. Then a 0.9 pp observed lift would be about 2.1 SE — starting to look meaningful. To make SE 10× smaller you need 10^2 = 100\times the users (200,000 per arm). Precision is bought with the square of the sample size, which is why detecting small lifts is expensive.
4. Why the CLT is the justification
Individual sign-ups are 0/1 Bernoulli outcomes — nothing like a Normal. The CLT says the average of many independent such outcomes is approximately Normal with variance p(1-p)/n, and that is the quantity we are comparing. Without the CLT there would be no reason to use Normal standard errors or the ±1.96 rule at all.
Share this question