Match a job Paths Subjects Questions Quizzes Pricing
Advanced Open Pro

Respond to a Stakeholder Reading a Short A/B Test as Failure

Two weeks into the launch A/B test, a stakeholder sees CTR down 3.1% relative and session length down 1.4% relative for the new long-term-optimizing ranker, and requests an immediate rollback, citing "the new model is clearly worse."

  1. Is the stakeholder's read defensible given only this two-week data? Explain what's missing from the picture.
  2. The full 4-week holdout later shows D30 retention +2.6 percentage points and sessions-per-user in weeks 3-4 up 4.8% relative, alongside the CTR and session-length declines. How would you present this result to the same stakeholder?
  3. Suppose instead both CTR and D30 retention had declined together. How would your recommendation change, and why?

Share this question

← Back to Case Study: Long-Term Engagement Recommender practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.