Paths Subjects Questions Quizzes Pricing Search
Advanced Open Pro

Canarying a Prompt Change with Rollback Criteria

A prompt change passed the CI regression gate: judge correctness, faithfulness, and format validity all held or improved against the versioned gold set. Before rolling it out to 100% of production traffic, design the canary plan.

  1. What percentage of traffic would you start with, and what would you watch during the canary window, beyond what CI already checked?
  2. Write two concrete rollback criteria (metric + threshold) and explain why each is set where it is, not tighter or looser.
  3. How long should the observation window be, and what's wrong with both "five minutes" and "promote to 100% only after CI passed, skip the canary entirely"?
  4. Who should be paged if a rollback threshold trips outside business hours, and what should happen automatically vs. require a human?

Share this question

← Back to LLM Observability and Evaluation practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.