Practice — Sequence Models: From RNNs to Attention (6 questions)

You train a character-level tanh RNN (hidden size 256) to predict whether a bracket opened at position 1 is closed correctly at position 50 (the 48 characters in between are noise). The training loss on short examples (length 5-10) goes down nicely, but on length-50 examples the model ignores the first bracket. A colleague proposes "a bigger learning rate" and another "just train longer".

1. Write the recurrence and derive \partial h_{50}/\partial h_1 as a product of Jacobians. Where do the recurrent weight matrix and the activation derivative appear?
2. Suppose the largest singular value of W_{hh} is 0.8 and the tanh slope is at most 1. Bound the norm of the gradient across 49 steps. What happens if the largest singular value is 1.2 instead? How do these two regimes show up in the loss curve and which one can be fixed with gradient clipping?
3. Explain why a bigger learning rate or longer training does not fix the first regime.
4. Name two architectural fixes and the property of each that helps, and one training-procedure limit (truncated BPTT) that can make the problem unlearnable even when the architecture is fine.

An LSTM-based code-completion model must track "am I currently inside a string literal, and which quote character opened it?" across lines of code. After training it handles 10-token strings but forgets the quote type in 100-token strings. You inspect it and find that the forget gate averages 0.55 on the memory unit that carries the quote type.

1. Write the LSTM equations and explain the role of the forget, input and output gates and of the candidate, using the string-literal unit as a running example. What should each gate do on the opening quote, on ordinary characters inside the string, and on the closing quote?
2. Why does the average forget value of 0.55 explain the failure? Compute the surviving fraction of the memory after 10 and after 100 steps at 0.55 and at 0.99.
3. Why is the cell-state path described as a "gradient highway", and what does the 0.55 imply about the gradient for this unit?
4. Give two concrete changes that you would make before collecting more data. Then say how the GRU relates and whether you would switch to it.