Advanced
Open
Pro
Feature Selection Inside vs Outside Cross-Validation
A colleague's workflow: (1) compute correlation of all 800 features with the target on the full training set, (2) keep the top 50 by absolute correlation, (3) run 5-fold cross-validation on a logistic regression using only those 50 features, reporting CV AUC = 0.88. On a genuinely held-out test set, AUC is 0.79.
- Explain why the CV AUC is optimistic, mechanistically.
- Rewrite the workflow so the reported CV score is honest.
- With 800 features and modest data, would you expect the gap to be worse with filter selection (correlation) or wrapper selection (e.g. forward selection using CV score as the search criterion)? Why?
Share this question