Intermediate
Open
Pro
Designing the Offline Evaluation Split
You are training a churn model for a subscription product. Churn is defined as "no login for 30 days". You have 18 months of event logs. A colleague proposes a random 80/20 train/test split over all (user, week) rows and reports AUC = 0.91.
- Explain two distinct ways the random split leaks information here.
- Design a split that avoids the leakage. Draw or describe the time windows and state the size of any gap.
- After switching to the correct split, AUC drops to 0.79. Is the model worse? What would you tell the product manager?
Share this question