Practice — Training Deep Networks: Optimizers, Schedules & Regularization (6 questions)

A teammate reviewing your optimizer code asks: "Why do we divide m_t and v_t by 1-\beta_1^t and 1-\beta_2^t? Isn't that just noise? PyTorch's Adam does it, but I'd like to delete it."

Use \beta_1 = 0.9, \beta_2 = 0.999, \epsilon = 10^{-8} and a single parameter whose first gradient is g_1 = 2.0.

1. Compute m_1 and v_1 from the recursions with m_0 = v_0 = 0. Then compute the size of the first update \eta\, m_1 / \sqrt{v_1} without bias correction, as a multiple of \eta.
2. Compute the corrected \hat m_1, \hat v_1 and the corrected first update. What general quantity does the uncorrected step equal?
3. How large is the correction factor 1 - \beta_2^t at t = 1000 and t = 3000? What does this tell you about when bias correction matters, and how does it relate to learning-rate warmup?

You are training a transformer. Config A uses Adam(weight_decay=0.01) (decay implemented as an L2 term added to the gradient). Config B uses AdamW(weight_decay=0.01). After tuning, B beats A and a colleague says "they're the same thing mathematically; it's seed noise."

1. Show when L2 regularization and weight decay are identical, and write the update rule for each under SGD.
2. Consider two parameters, both at \theta = 0.5 with \lambda = 0.01, whose steady gradient magnitudes are 1.0 and 0.001. Compare how much each is decayed per step (relative to \eta) under Adam+L2 and under AdamW.
3. Name two practical consequences for hyperparameter tuning, and list which parameters you would exclude from decay in either case.