Regularization: L1, L2 and Elastic Net
A linear model with p features has p coefficients it can move freely. When p is large relative to the number of rows, when features are correlated, or when there is noise in the target, ordinary least squares (OLS) will happily set some coefficients to enormous values with opposite signs so that they cancel on the training rows — a fit that looks great in-sample and falls apart the moment a new row arrives. Regularization is the fix: add a penalty on the size of the coefficients to the training objective, so the model has to pay for every unit of coefficient magnitude and only spends where the data justifies it.
This is one of the most heavily examined topics in data-science interviews because it connects everything: linear algebra (the closed-form ridge solution), geometry (why L1 gives sparsity), Bayesian statistics (priors), optimisation (soft-thresholding, weight decay) and practical model selection (choosing \lambda by cross-validation). "Explain lasso vs ridge" is a first-round question; "why does lasso pick one of two correlated features while ridge splits the weight?" is the follow-up that separates candidates who memorised from candidates who understand.
This subject covers the penalties themselves and their behaviour. The mechanics of how you run cross-validation to select the penalty strength — folds, leakage, nested CV — are covered in the Bias–Variance Trade-off & Cross-Validation subject; here we only use CV as a black box that returns a validation score for each \lambda.
What Overfitting Looks Like in Coefficient Space
Take a regression with two features that are almost identical (correlation 0.9). Both are mildly predictive of the target. OLS solves for the coefficient vector that minimises training squared error, and because the two columns are nearly collinear, the loss surface is a long, flat valley: many very different (\beta_1, \beta_2) pairs give almost the same training error. OLS picks the exact bottom of that valley — which is determined by noise.
Coefficient space for two correlated features (contours = training RSS)
β2
▲
│ ╱╱╱╱╱╱╱╱╱╱╱ Long flat valley: RSS barely
│ ╱╱╱ ╱╱╱╱╱ ╱╱╱ changes as you slide along it.
│ ╱╱╱ ╱╱╱ ★ ╱╱╱ ╱╱╱ ★ = OLS solution — sits wherever
│ ╱╱╱ ╱╱╱ ╱╱╱ ╱╱╱ ╱╱╱ noise pushed it, e.g. (1.47, -0.53)
│╱╱╱ ╱╱╱ ╱╱╱ ╱╱╱ ╱╱╱
────┼───────────────────────────────▶ β1
│ ╱╱╱ ╱╱╱ ╱╱╱ ╱╱╱ ╱╱╱
│
Symptoms you actually see in a fitted model:
| Symptom | Why it happens |
|---|---|
| Coefficients with huge magnitude and opposite signs on correlated features | They cancel each other on training rows; tiny input perturbations flip predictions |
| Coefficients change wildly between bootstrap resamples | The valley floor moves with the noise |
| Train R^2 near 1, validation R^2 much lower | The model has fitted noise, not signal |
| Condition number of X^\top X very large | Near-singular design matrix; the inverse amplifies noise |
Every regulariser you'll meet is a way of saying: among all points in that valley, prefer the one closest to the origin.