Paths Subjects Questions Quizzes Pricing Search
Intermediate Open Pro

Designing the Offline Evaluation Split

You are training a churn model for a subscription product. Churn is defined as "no login for 30 days". You have 18 months of event logs. A colleague proposes a random 80/20 train/test split over all (user, week) rows and reports AUC = 0.91.

  1. Explain two distinct ways the random split leaks information here.
  2. Design a split that avoids the leakage. Draw or describe the time windows and state the size of any gap.
  3. After switching to the correct split, AUC drops to 0.79. Is the model worse? What would you tell the product manager?

Share this question

← Back to Model Training & Experimentation at Scale practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.