Advanced
Open
Pro
Canarying a Prompt Change with Rollback Criteria
A prompt change passed the CI regression gate: judge correctness, faithfulness, and format validity all held or improved against the versioned gold set. Before rolling it out to 100% of production traffic, design the canary plan.
- What percentage of traffic would you start with, and what would you watch during the canary window, beyond what CI already checked?
- Write two concrete rollback criteria (metric + threshold) and explain why each is set where it is, not tighter or looser.
- How long should the observation window be, and what's wrong with both "five minutes" and "promote to 100% only after CI passed, skip the canary entirely"?
- Who should be paged if a rollback threshold trips outside business hours, and what should happen automatically vs. require a human?
Share this question