Advanced
Open
Pro
Grouped and Temporal Leakage in a Churn Model
A subscription company builds a churn model. The dataset has one row
per (user, week) — each user contributes roughly 20 weekly rows over
five months, with a label "churned in the next 30 days". Data scientist
A uses KFold(shuffle=True) and reports 0.93 AUC in CV. In production,
the model scores 0.74 AUC on the next month of real traffic.
- Identify the two distinct leakage mechanisms most likely at play here (there is more than one), and explain the mechanism by which each inflates the CV score.
- Propose a splitting strategy that would have caught this before deployment, and describe exactly how the folds should be constructed.
- A colleague argues "just use
GroupKFoldonuser_idand the problem is solved." Is that sufficient on its own? Why or why not?
Share this question