Derive Why a Vanilla RNN Cannot Learn a 50-Step Dependency
You train a character-level tanh RNN (hidden size 256) to predict whether a bracket opened at position 1 is closed correctly at position 50 (the 48 characters in between are noise). The training loss on short examples (length 5-10) goes down nicely, but on length-50 examples the model ignores the first bracket. A colleague proposes "a bigger learning rate" and another "just train longer".
- Write the recurrence and derive \partial h_{50}/\partial h_1 as a product of Jacobians. Where do the recurrent weight matrix and the activation derivative appear?
- Suppose the largest singular value of W_{hh} is 0.8 and the tanh slope is at most 1. Bound the norm of the gradient across 49 steps. What happens if the largest singular value is 1.2 instead? How do these two regimes show up in the loss curve and which one can be fixed with gradient clipping?
- Explain why a bigger learning rate or longer training does not fix the first regime.
- Name two architectural fixes and the property of each that helps, and one training-procedure limit (truncated BPTT) that can make the problem unlearnable even when the architecture is fine.
1. The product of Jacobians. With h_t = \tanh(W_{xh} x_t + W_{hh} h_{t-1} + b), one step has Jacobian
(the diagonal is the tanh derivative evaluated at the pre-activation; the matrix is the same W_{hh} at every step because of parameter sharing). By the chain rule over the unrolled graph (BPTT),
a product of 49 matrices. The loss at step 50 depends on the bracket at step 1 only through this product, so it is the entire learning signal for that dependency.
2. The bound. The norm of a product is at most the product of the norms. With \lVert W_{hh} \rVert_2 = 0.8 and tanh slope at most 1, each factor has norm at most 0.8, so \lVert \partial h_{50}/\partial h_1 \rVert \le 0.8^{49} \approx 1.8 \times 10^{-5} (and the saturated tanh units make the true value smaller still). The signal from step 50 to step 1 is five orders of magnitude below that from step 49 to step 48: vanishing gradients. With a largest singular value of 1.2 the product can grow like 1.2^{49} \approx 7{,}600 (it is only an upper bound, because tanh slopes below 1 damp it, but the growth is exponential once the recurrent weights dominate): exploding gradients.
In the loss curve, exploding gradients appear as sudden spikes and NaN (huge steps), and gradient-norm clipping fixes them by rescaling the whole gradient to a maximum norm \tau while keeping its direction. Vanishing gradients appear as a quiet failure: training looks healthy because the short examples are learned, but the long dependency receives essentially no gradient. Clipping cannot help because it only scales gradients down; there is no signal to rescale.
3. Why learning rate and training time do not help. The gradient wrt the early step is not merely small, it is a systematically shrunken version of the gradient that is learnable at the late steps. Scaling every gradient by a bigger learning rate also scales the near-term terms (which then diverge) while the long-range term is still 10^{5} times smaller than its neighbours, so the update is dominated by short-range structure. More epochs just keep optimizing that structure. The optimization problem itself is ill-conditioned along the long-range direction.
4. Fixes.
- LSTM / GRU: the additive cell update c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t gives a direct gradient path \partial c_t/\partial c_{t-1} = \operatorname{diag}(f_t) with no weight matrix and no tanh derivative; with the forget gate near 1 (so initialize its bias to about 1) the gradient passes through many steps. With f = 0.95, 0.95^{49} = 0.08 instead of 10^{-5}.
- Attention (or a Transformer): every position is connected to every other in one layer, so the path length is O(1) instead of 49 multiplications.
- Others: orthogonal initialization of W_{hh}, normalization, or skip connections across time.
Truncated BPTT limits the unroll to K steps. If K = 20 then gradient never flows from step 50 to step 1, so no architecture can learn the dependency from gradients alone; choose K larger than the dependency length you need (here at least 50), or use a model whose path length does not depend on K.
Share this question