Regularisation Beyond Linear Models

Your team runs three different model families and asks you to unify their view of "regularisation":

• A PyTorch MLP trained with Adam and weight_decay=1e-4.
• The same MLP trained with SGD and an explicit + 0.5 * λ * ||w||² term added to the loss.
• A LightGBM model with lambda_l2=10, lambda_l1=1, min_data_in_leaf=100.

1. Show mathematically why weight decay under plain SGD is equivalent to L2 regularisation, and explain why the equivalence breaks under Adam. What optimiser variant fixes it?
2. What do lambda_l2 and lambda_l1 act on in a gradient-boosted tree, and how do their effects mirror ridge and lasso? What is min_data_in_leaf's role in this picture?
3. Early stopping is used in all three setups. Explain in what sense it is itself a regulariser and, for a linear model trained by gradient descent, what it approximates.