Advanced
Open
Pro
Design a Model-Quality Gate That Would Have Caught a Slice Regression
A PR changes how a support_tickets_last_30d feature handles missing
values. Aggregate offline AUC barely moves (0.841 → 0.839), so a
naive "block if AUC drops more than 1 point" gate would pass this PR.
The actual effect is a 6-point recall drop specifically on the
"low-engagement, high-churn-risk" customer segment.
- Design a model-quality gate that would have caught this, and explain specifically why the naive aggregate-AUC gate missed it.
- What's the trade-off of adding many per-slice checks to a CI gate, and how would you decide which slices are worth gating on?
- Suppose the per-slice check is in place but flags a different PR for a segment that turns out, on inspection, to be genuinely tiny (0.1% of traffic) and business-irrelevant. What does this suggest about how the gate should be tuned going forward?
Share this question