Sizing a Checkout Experiment
Your team wants to test a redesigned checkout button. Current checkout conversion is 4.0%. Product says they would ship the change only if it lifts conversion by at least 10% relative. About 8,000 eligible users reach checkout per day, and you will split traffic 50/50.
- Using \alpha = 0.05 (two-sided) and 80% power, estimate the sample size per arm and the total number of days the test needs.
- Product asks whether you can finish in 5 days. What is the smallest relative lift you could detect in that time, and how would you explain the trade-off?
- Even if the sample size were reached in 3 days, would you stop then? Why or why not?
1. Sample size and duration
Absolute MDE: \delta = 0.10 \times 0.04 = 0.004 (4.0% → 4.4%). Using n \approx (z_{1-\alpha/2} + z_{1-\beta})^2 \cdot 2p(1-p) / \delta^2 with z = 1.96 and 0.84:
n \approx 7.85 \times 2 \times 0.04 \times 0.96 / 0.004^2 = 7.85 \times 0.0768 / 0.000016 \approx 37{,}700 per arm.
About 75,000 users total. At 8,000 per day that is roughly 9.5 days, so plan for 14 days (two full weekly cycles).
2. Five-day constraint
Five days gives 40,000 users, 20,000 per arm. Sample size scales with 1/\delta^2, so the detectable effect scales with 1/\sqrt{n}: \delta_{5d} \approx 0.004 \times \sqrt{37{,}700 / 20{,}000} \approx 0.0055, i.e. about a 14% relative lift. Explain it as: "In five days we can reliably see a 14% lift or larger. If the true effect is the 10% you would ship on, we have well under 80% power — a null result would tell us almost nothing, and any win we do see will be an over-estimate (winner's curse)." Offer alternatives: CUPED using pre-period behaviour, a more sensitive metric, or accepting the longer test.
3. Stopping at day 3
No. Three days covers only part of the weekly cycle; weekday and weekend shoppers differ in intent and device mix, so the estimated effect applies to a biased population. Also, cookie/identity churn and novelty effects are not visible yet. Run at least one full week, ideally two, even when the sample size is reached earlier. The one thing you should look at from day 1 is health: SRM and assignment logging.
Share this question