Explain Adam's Bias Correction With Numbers
A teammate reviewing your optimizer code asks: "Why do we divide m_t and v_t by 1-\beta_1^t and 1-\beta_2^t? Isn't that just noise? PyTorch's Adam does it, but I'd like to delete it."
Use \beta_1 = 0.9, \beta_2 = 0.999, \epsilon = 10^{-8} and a single parameter whose first gradient is g_1 = 2.0.
- Compute m_1 and v_1 from the recursions with m_0 = v_0 = 0. Then compute the size of the first update \eta\, m_1 / \sqrt{v_1} without bias correction, as a multiple of \eta.
- Compute the corrected \hat m_1, \hat v_1 and the corrected first update. What general quantity does the uncorrected step equal?
- How large is the correction factor 1 - \beta_2^t at t = 1000 and t = 3000? What does this tell you about when bias correction matters, and how does it relate to learning-rate warmup?
1. Uncorrected step
m_1 = 0.9 \cdot 0 + 0.1 \cdot 2.0 = 0.2 and v_1 = 0.999 \cdot 0 + 0.001 \cdot 2.0^2 = 0.004. Without correction the update is
so the very first step would be about three times larger than intended.
2. Corrected step
\hat m_1 = 0.2 / (1 - 0.9) = 2.0 and \hat v_1 = 0.004 / (1 - 0.999) = 4.0, so \sqrt{\hat v_1} = 2.0 and the update is \eta \cdot 2.0/2.0 = \eta, i.e. exactly \eta\operatorname{sign}(g_1). In general the uncorrected step equals \frac{1-\beta_1}{\sqrt{1-\beta_2}} = 0.1/0.0316 = 3.16 times the corrected step at t = 1: both moments are biased toward zero because they start at zero, but v (which has the much larger \beta_2) is biased far more, so \sqrt{v} is underestimated more than m is, and the ratio is inflated.
3. When it matters
1 - 0.999^{1000} = 0.632 and 1 - 0.999^{3000} = 0.950. The v estimate is still noticeably biased (by a factor of 1/0.632 = 1.58 in v, or 1.26 in \sqrt{v}) a thousand steps in, and only approximately unbiased after a few thousand. The first moment is unbiased much sooner: 1 - 0.9^{44} = 0.990. So the correction is essential in the first steps (and for \beta_2 closer to 1) and negligible later. The teammate's instinct that it "does nothing" late in training is right, but deleting it makes the first steps 3x too big, which is a classic source of early loss spikes.
The relation to warmup: bias correction fixes the systematic error in the early moment estimates, but even the corrected \hat v_t is computed from just a few gradients, so it is a high-variance estimate of the true second moment and the adaptive step sizes are erratic. Warmup keeps \eta small until \hat v stabilizes. They are complementary: correction removes bias, warmup tolerates variance.
Share this question