Match a job Paths Subjects Questions Quizzes Pricing
Intermediate Open Pro

Calibrate a Model-Quality Gate's Tolerance Band

A team's churn model is retrained weekly. Looking at the last 12 weekly retrains (same code, rolling data window, no intentional changes), the champion's AUC on a fixed, time-forward holdout has ranged from 0.851 to 0.861, with a mean of 0.856 and a standard deviation of about 0.003.

  1. Propose a specific numeric tolerance band for the model-quality gate, and explain your reasoning.
  2. A new challenger, produced by a PR that adds two new features, scores 0.849 on the same holdout. Does it pass your gate? What would you do next?
  3. A different challenger, from a PR that only fixes a typo in a log message (no logic change), scores 0.883 — 2.7 points above the 12-week maximum. What does this suggest, and how does your gate design help catch it?

Share this question

← Back to CI/CD for ML & Data Code practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.