Practice — A/B Testing & Online Experimentation (6 questions)

Your team wants to test a redesigned checkout button. Current checkout conversion is 4.0%. Product says they would ship the change only if it lifts conversion by at least 10% relative. About 8,000 eligible users reach checkout per day, and you will split traffic 50/50.

1. Using \alpha = 0.05 (two-sided) and 80% power, estimate the sample size per arm and the total number of days the test needs.
2. Product asks whether you can finish in 5 days. What is the smallest relative lift you could detect in that time, and how would you explain the trade-off?
3. Even if the sample size were reached in 3 days, would you stop then? Why or why not?

You launch a 30-day A/B test at the standard 5% significance level. Every morning you check the dashboard, and you'll ship the variant the first day it shows p < 0.05.

If the variant actually does nothing, what's the real probability you'll declare a winner by day 30?

Explain why it isn't 5%, and name a testing approach that makes peeking safe.