Paths Subjects Questions Quizzes Pricing Search
Advanced Open Pro

Grouped and Temporal Leakage in a Churn Model

A subscription company builds a churn model. The dataset has one row per (user, week) — each user contributes roughly 20 weekly rows over five months, with a label "churned in the next 30 days". Data scientist A uses KFold(shuffle=True) and reports 0.93 AUC in CV. In production, the model scores 0.74 AUC on the next month of real traffic.

  1. Identify the two distinct leakage mechanisms most likely at play here (there is more than one), and explain the mechanism by which each inflates the CV score.
  2. Propose a splitting strategy that would have caught this before deployment, and describe exactly how the folds should be constructed.
  3. A colleague argues "just use GroupKFold on user_id and the problem is solved." Is that sufficient on its own? Why or why not?

Share this question

← Back to Bias–Variance Trade-off & Cross-Validation practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.