Diagnosing a Model From Learning and Validation Curves
You train a gradient boosting classifier on 50,000 rows. Training AUC
is 0.995, validation AUC is 0.81. You plot a learning curve (error vs
training-set size) and both the training and validation curves are
still converging toward each other as you add more rows, with a
visible gap remaining at the full dataset size. You also plot a
validation curve sweeping max_depth from 2 to 12, and validation AUC
peaks at max_depth=4 then declines.
- What do the two curves individually tell you, and what is the combined diagnosis?
- Rank the following interventions by expected usefulness here, and
justify the ranking: (a) collect more training rows, (b) reduce
max_depthto 4, (c) add L2 regularisation on leaf weights, (d) engineer better features. - After applying your top interventions, validation AUC rises to 0.87 but training AUC drops to 0.90. Is this an improvement? Explain.
1. Reading the curves
The learning curve shows a persistent, still-closing gap between train
and validation error even at full data size — the textbook signature
of high variance / overfitting: the model is complex enough to fit
idiosyncrasies of the training sample, and more rows are still helping
because they are diluting that noise, but the gap hasn't closed yet.
The validation curve confirms the same diagnosis from a different
angle: past max_depth=4, validation AUC falls while training AUC
keeps climbing — classic overfitting as complexity increases, with the
current model (uncapped or deep) sitting well past the sweet spot.
Combined diagnosis: this is a variance problem, not a bias problem —
the model is expressive enough (arguably too expressive); it needs to
be constrained, not made richer.
2. Ranking the interventions
Most useful first: (b) reduce max_depth to 4 — the validation
curve directly identifies this as the complexity level with the best
out-of-sample score; it's the most targeted fix and free (no new data
or features needed). (c) add L2 regularisation on leaf weights —
a complementary variance reducer; combine with (b) rather than instead
of it. (a) collect more training rows — the learning curve shows
this is still helping (curves are converging), so it is genuinely
useful, but it's usually the slowest and most expensive lever compared
to changing a hyperparameter, and the gap suggests structural
overfitting that more data alone would need a lot of volume to fully
close. (d) engineer better features — this targets bias, not the
diagnosed variance problem; it might help in general but is not what
the evidence here calls for, so it ranks last for this specific
diagnosis (though worth doing eventually for its own sake).
3. Is dropping training AUC from 0.995 to 0.90 an improvement?
Yes, decisively. Training AUC was never the target — it was inflated by overfitting and was never going to reflect production performance. Validation AUC (the honest out-of-sample estimate) rose from 0.81 to 0.87, a real, meaningful gain. Judging model quality by training metrics is exactly the mistake this subject warns against; the shrinking train/validation gap (0.995→0.81 gap of 0.185, now 0.90→0.87 gap of 0.03) is itself evidence of a healthier bias-variance balance, independent of the validation number improving.
Share this question