Match a job Paths Subjects Questions Quizzes Pricing
Advanced Open Pro

Design a Model-Quality Gate That Would Have Caught a Slice Regression

A PR changes how a support_tickets_last_30d feature handles missing values. Aggregate offline AUC barely moves (0.841 → 0.839), so a naive "block if AUC drops more than 1 point" gate would pass this PR. The actual effect is a 6-point recall drop specifically on the "low-engagement, high-churn-risk" customer segment.

  1. Design a model-quality gate that would have caught this, and explain specifically why the naive aggregate-AUC gate missed it.
  2. What's the trade-off of adding many per-slice checks to a CI gate, and how would you decide which slices are worth gating on?
  3. Suppose the per-slice check is in place but flags a different PR for a segment that turns out, on inspection, to be genuinely tiny (0.1% of traffic) and business-irrelevant. What does this suggest about how the gate should be tuned going forward?

Share this question

← Back to Case Study: Churn Model, End-to-End MLOps practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.