Match a job Paths Subjects Questions Quizzes Pricing
Advanced Open Pro

Decide Whether a Retrained Challenger Needs Human Review

The retrained challenger (post price-increase) scores AUC 0.858 versus the champion's 0.836 on the fresh time-forward holdout — a 2.2-point gain. The team's hybrid gating policy auto-promotes within a tolerance band and routes to human review outside it.

  1. Should this specific promotion auto-promote or route to review? Justify using the specifics of this scenario, not just the size of the number.
  2. A reviewer approves it. What, specifically, should be recorded at the moment of promotion so a stakeholder asking "why did the flagged-customer mix change in October" gets a fast, sourced answer three months later?
  3. Contrast this with a hypothetical second scenario: a routine weekly retrain (no known drift event) shows an unexplained 2.2-point AUC jump. Would your recommendation differ, and why?

Share this question

← Back to Case Study: Churn Model, End-to-End MLOps practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.