Advanced
Open
Pro
Respond to a Stakeholder Reading a Short A/B Test as Failure
Two weeks into the launch A/B test, a stakeholder sees CTR down 3.1% relative and session length down 1.4% relative for the new long-term-optimizing ranker, and requests an immediate rollback, citing "the new model is clearly worse."
- Is the stakeholder's read defensible given only this two-week data? Explain what's missing from the picture.
- The full 4-week holdout later shows D30 retention +2.6 percentage points and sessions-per-user in weeks 3-4 up 4.8% relative, alongside the CTR and session-length declines. How would you present this result to the same stakeholder?
- Suppose instead both CTR and D30 retention had declined together. How would your recommendation change, and why?
Share this question