Why AdamW, Not Adam With L2?
You are training a transformer. Config A uses Adam(weight_decay=0.01)
(decay implemented as an L2 term added to the gradient). Config B uses
AdamW(weight_decay=0.01). After tuning, B beats A and a colleague
says "they're the same thing mathematically; it's seed noise."
- Show when L2 regularization and weight decay are identical, and write the update rule for each under SGD.
- Consider two parameters, both at \theta = 0.5 with \lambda = 0.01, whose steady gradient magnitudes are 1.0 and 0.001. Compare how much each is decayed per step (relative to \eta) under Adam+L2 and under AdamW.
- Name two practical consequences for hyperparameter tuning, and list which parameters you would exclude from decay in either case.
1. SGD: the two coincide
L2 regularization minimizes L + \frac{\lambda}{2}\lVert\theta\rVert^2, adding \lambda\theta to the gradient. The SGD update is
which is exactly "decay the weights by the factor (1-\eta\lambda), then take the gradient step", i.e. weight decay. For SGD the two are the same thing (with the coefficient defined this way).
2. Adam breaks the equivalence
With Adam+L2 the penalty gradient enters the moment estimates, so the decay's effective size is divided by \sqrt{\hat v}, which is about the magnitude of the total gradient, \lvert g + \lambda\theta\rvert:
- Parameter with \lvert g \rvert = 1.0: total \approx 1.005; decay contribution \approx \eta \cdot 0.005/1.005 \approx 0.005\,\eta.
- Parameter with \lvert g \rvert = 0.001: total \approx 0.006; decay contribution \approx \eta \cdot 0.005/0.006 \approx 0.83\,\eta.
The second parameter is decayed about 167x harder, purely because its gradients are small. Under AdamW the decay is \eta\lambda\theta = \eta \cdot 0.005 for both, applied outside the adaptive normalization. So the colleague is wrong: Adam+L2 regularizes parameters unevenly in a way unrelated to how much each needs it (it nearly zeroes weights that rarely receive gradient and barely regularizes those with big gradients), and AdamW shrinks them all proportionally.
3. Consequences
- In AdamW the optimum for \lambda is roughly independent of \eta (the shrink rate per step is \eta\lambda and you can reason about it), whereas under Adam+L2 the interaction with the adaptive denominator makes the two hyperparameters entangled and the best \lambda depends on gradient scale, so searches are larger and results do not transfer across model sizes.
- AdamW's decay is uniform and predictable: with \eta = 3\times10^{-4} and \lambda = 0.1, each step shrinks weights by 0.003%, about 95% over 100k steps if the gradient did not push back, a number you can sanity-check.
In both cases exclude 1-D parameters from decay: biases and
LayerNorm/BatchNorm gains and offsets (and usually embeddings), using
separate parameter groups with weight_decay=0. They set activation
scale rather than function complexity, and decaying them hurts.
Share this question