Lasso vs Ridge on Correlated Features
You are modelling customer churn with a linear model. Among your 40
features there is a block of six that are strongly correlated with each
other (pairwise correlation 0.85–0.95): sessions_7d, sessions_14d,
sessions_30d, sessions_60d, sessions_90d, sessions_180d. All six
are individually predictive of churn.
You fit an unpenalised model, a Ridge model and a Lasso model, tuning \lambda by 5-fold CV in each case.
- Describe what you expect the six coefficients to look like under each of the three models, and explain the mechanism behind each pattern.
- A colleague reruns the Lasso on a different random 5-fold split and gets a different member of the block selected. Is the model broken? What would you recommend instead, and why?
- If your goal is interpretation ("which activity window matters most?"), which of the three models is the most misleading, and why?
1. Expected coefficient patterns
Unpenalised OLS: the six coefficients will be large in magnitude and have mixed signs — e.g. +2.1, −1.8, +1.5, −0.9, … — even though every feature is positively related to churn. The design matrix has a near-singular direction (the differences between the six columns), and OLS amplifies whatever noise happens to fall along that direction. The coefficients will also be highly unstable under resampling.
Ridge: the six coefficients will be small, similar in size and share the same sign. Ridge shrinks each principal direction by d_j^2/(d_j^2+\lambda): the shared "how much does session activity matter" direction has a large eigenvalue and is barely shrunk, while the "which window gets the credit" directions have tiny eigenvalues and are crushed. The total effect of the block is preserved and split roughly evenly. No coefficient is exactly zero.
Lasso: typically one or two of the six will be non-zero and the rest exactly zero. The L1 penalty applies a constant pull toward zero, and once one member of the block explains most of the shared signal, the marginal gain of the others is below the threshold and they are pinned at zero. Which member survives is decided by small differences in correlation with the target — effectively by noise.
2. Different feature selected on a different split
No, this is expected lasso behaviour with correlated blocks: the
selection within a highly correlated group is essentially arbitrary,
so a different fold assignment changes which twin wins. Recommend
Elastic Net with, say, l1_ratio around 0.5 tuned by CV. The L2
component gives the grouping effect: correlated features enter and
leave the model together with similar coefficients, while the L1
component still zeroes out the genuinely irrelevant features elsewhere
in the 40. Alternatively, engineer a single summary feature for the
block (e.g. one representative window or a PCA score) before fitting.
3. Most misleading for interpretation
OLS is the most misleading: the large opposite-signed coefficients
would suggest, for instance, that more 14-day activity increases churn
while more 7-day activity decreases it, which is an artefact of
collinearity, not a real effect. Lasso is misleading in a subtler way —
it would tell you "only sessions_30d matters", when in fact any of
the six carries the same information. Ridge (or Elastic Net) gives the
most honest picture: the block collectively matters, and no individual
window is singled out because the data cannot distinguish them.
Share this question