Advanced
Open
Pro
Decide Whether a Retrained Challenger Needs Human Review
The retrained challenger (post price-increase) scores AUC 0.858 versus the champion's 0.836 on the fresh time-forward holdout — a 2.2-point gain. The team's hybrid gating policy auto-promotes within a tolerance band and routes to human review outside it.
- Should this specific promotion auto-promote or route to review? Justify using the specifics of this scenario, not just the size of the number.
- A reviewer approves it. What, specifically, should be recorded at the moment of promotion so a stakeholder asking "why did the flagged-customer mix change in October" gets a fast, sourced answer three months later?
- Contrast this with a hypothetical second scenario: a routine weekly retrain (no known drift event) shows an unexplained 2.2-point AUC jump. Would your recommendation differ, and why?
Share this question