Intermediate
Open
Pro
Metric Moved but Not Significant, and a Winning Segment
An onboarding experiment finishes at its planned sample size. Primary metric (7-day activation) shows +2.1% relative lift, 95% CI [-0.9%, +5.1%], p = 0.17. The pre-registered MDE was 5%. An analyst then slices by 16 country × platform combinations and finds that iOS users in Germany show +18% (p = 0.01).
- How do you interpret the primary result? Should the test be extended?
- How do you interpret the Germany-iOS finding? Quantify the risk.
- What would you recommend to the team?
Share this question