Canary Sample Size and a Premature Rollback

A recommendation team runs a canary at 2% of traffic for a challenger ranking model. After 20 minutes, click-through rate (CTR) on the canary slice is 1.9% versus 2.3% on the champion slice — a 17% relative drop. An on-call engineer, worried, wants to roll back immediately.

1. The platform serves roughly 50,000 sessions/hour in total. Given the 2% canary slice and the 20-minute window, roughly how many canary sessions have been observed, and why does that matter for whether 1.9% vs 2.3% is a meaningful signal or noise?
2. What would you want to know or check before deciding whether to roll back, hold, or continue ramping?
3. Propose a concrete rule the team should have agreed on before the canary started, so this situation doesn't turn into an ad hoc debate at decision time.