Finding and Fixing a Leaky Preprocessing Pipeline

Review this notebook cell:

from sklearn.preprocessing import StandardScaler
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, StratifiedKFold

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

selector = SelectKBest(f_classif, k=20)
X_selected = selector.fit_transform(X_scaled, y)

clf = LogisticRegression(C=1.0)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)
scores = cross_val_score(clf, X_selected, y, cv=cv, scoring="roc_auc")
print(scores.mean())   # 0.91

The dataset has 300 features and 1,200 rows, and roughly 250 of those features are pure random noise unrelated to y (the data scientist confirms this from the data generation process).

1. Identify every leak in this code and explain the mechanism for each.
2. Given that ~250 of 300 features are pure noise, what would you expect the honest CV AUC to be if the leaks are fixed, roughly, and why does that matter for interpreting the 0.91?
3. Rewrite the cell so that cross_val_score reports an honest number.
4. Would nested CV be necessary here in addition to your fix? Why or why not?