Sequence Models: From RNNs to Attention
Why recurrence was invented, why gates fixed it, and why attention replaced it
Understand sequence modelling end to end and be able to defend each step in an interview: tensor shapes (batch, time, features) with padding, masking and packed sequences, the many-to-one / one-to-many / seq2seq taxonomy, the vanilla RNN with parameter sharing across time, BPTT and truncated BPTT, why gradients vanish or explode through time (a product of Jacobians, with a verified experiment), LSTM and GRU gate equations, the cell state as a gradient highway and the forget-bias initialization trap, bidirectional and stacked variants with exact parameter counts, training details (teacher forcing and exposure bias, masked losses, gradient clipping, bucketing), the seq2seq fixed-vector bottleneck, Bahdanau and Luong attention and the query/key/value framing, self-attention and why Transformers won (path length, parallelism, the O(n^2) cost, and where RNNs and state-space models like Mamba remain), decoding (greedy, beam search with length penalty, temperature, top-k and top-p sampling), and a worked example with a hand-computed 3-token dot-product attention step, a one-unit LSTM step and verified LSTM parameter counts.
Practice questions (6)
-
View →
Derive Why a Vanilla RNN Cannot Learn a 50-Step Dependency
Intermediate · Free -
View →
Explain Each LSTM Gate With a Concrete Example and Diagnose a Forgetful Model
Intermediate · Free -
View →
RNN vs Transformer for Streaming Speech Recognition on a Phone
Advanced -
View →
Compute One Attention Step by Hand and Reason About Masking and Scaling
Intermediate -
View →
Exposure Bias: A Summarizer That Validates Well and Rambles in Production
Intermediate