Match a job Paths Subjects Questions Quizzes Pricing
Advanced Open Pro

Design the Launch Experiment for the RL Notification Policy

Your RL notification policy has cleared the off-policy evaluation gate and a two-week capped canary with no guardrail alarms. You now need to design the randomized experiment that decides whether to ship it to 100% of traffic.

  1. Specify the randomization unit, the primary metric, and the minimum duration, and justify each choice against this system's specific failure modes.
  2. A stakeholder proposes reading the experiment's open-rate delta at day 7 and shipping early if it's clearly positive, to "move faster since the canary already looked safe." Explain concretely what this risks, using the reward-hacking mechanism from earlier in this case study.
  3. The topline 30-day retention result comes back +1.8%, which clears the pre-registered bar. Before shipping, what additional check would you insist on, and why might a topline win still be the wrong decision?

Share this question

← Back to Case Study: Notification Timing & Content Optimization practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.