Advanced
Open
Pro
Design the Launch Experiment for the RL Notification Policy
Your RL notification policy has cleared the off-policy evaluation gate and a two-week capped canary with no guardrail alarms. You now need to design the randomized experiment that decides whether to ship it to 100% of traffic.
- Specify the randomization unit, the primary metric, and the minimum duration, and justify each choice against this system's specific failure modes.
- A stakeholder proposes reading the experiment's open-rate delta at day 7 and shipping early if it's clearly positive, to "move faster since the canary already looked safe." Explain concretely what this risks, using the reward-hacking mechanism from earlier in this case study.
- The topline 30-day retention result comes back +1.8%, which clears the pre-registered bar. Before shipping, what additional check would you insist on, and why might a topline win still be the wrong decision?
Share this question