Intermediate
Open
Pro
Choosing k and the Right Splitter
You are advising three teams:
- Team 1: 800 rows, a ridge regression model, wants the most reliable possible estimate of test RMSE and has cheap compute.
- Team 2: 2 million clickstream rows spanning 18 months, building a model to predict next-week purchases; the data has a clear seasonal trend and a known regime change (a pricing policy changed month 10).
- Team 3: 50,000 rows of medical images, ~8 images per patient, building a diagnostic classifier; new patients arrive continuously in production and no patient in training will ever reappear.
- For each team, recommend a specific CV strategy (splitter, k if applicable, window type if relevant) and justify it against their data structure.
- For Team 2 specifically, explain why a random 80/20 split would give a misleadingly good offline number, tying your answer to the pricing regime change.
- For Team 3, what happens to the validation AUC if they accidentally
use plain
KFoldinstead of your recommendation, and in which direction is the bias?
Share this question