Paths Subjects Questions Quizzes Pricing Search
Advanced Open Pro

Nested CV vs a Single Test Set

A team runs GridSearchCV with 5-fold CV over 150 combinations of max_depth, num_leaves and learning_rate for a LightGBM model, finds the best combination scores 0.912 AUC in that search, and reports "our model achieves 0.912 AUC" in a slide deck.

  1. Explain precisely why 0.912 is likely an overestimate of the model's true out-of-sample AUC, tying your answer to the number of combinations searched.
  2. Describe two different fixes and the trade-off between them: (a) a single held-out test set, (b) nested cross-validation. When would you prefer one over the other?
  3. Suppose the team implements nested CV and the five outer folds give AUCs of 0.87, 0.90, 0.85, 0.89, 0.88, with a different best (max_depth, num_leaves, learning_rate) winning in three of the five outer folds. Is that a problem? What should the team report, and what should they do to get the final deployed model?

Share this question

← Back to Bias–Variance Trade-off & Cross-Validation practice

We use cookies for product analytics to improve OmniAtlas. See our Privacy Policy.