Advanced
Open
Pro
Nested CV vs a Single Test Set
A team runs GridSearchCV with 5-fold CV over 150 combinations of
max_depth, num_leaves and learning_rate for a LightGBM model,
finds the best combination scores 0.912 AUC in that search, and reports
"our model achieves 0.912 AUC" in a slide deck.
- Explain precisely why 0.912 is likely an overestimate of the model's true out-of-sample AUC, tying your answer to the number of combinations searched.
- Describe two different fixes and the trade-off between them: (a) a single held-out test set, (b) nested cross-validation. When would you prefer one over the other?
- Suppose the team implements nested CV and the five outer folds give
AUCs of 0.87, 0.90, 0.85, 0.89, 0.88, with a different best
(max_depth, num_leaves, learning_rate)winning in three of the five outer folds. Is that a problem? What should the team report, and what should they do to get the final deployed model?
Share this question