Sequence Models: From RNNs to Attention
This subject assumes Neural Networks & Backpropagation (the chain rule, vanishing and exploding gradients, initialization), Training Deep Networks: Optimizers, Schedules & Regularization (gradient clipping, residual connections, normalization) and The PyTorch Training Loop from Scratch; it covers what is specific to ordered data: recurrence, gating, encoder-decoder models, attention and decoding. The full Transformer architecture (multi-head attention, positional encodings, KV cache) is in Transformers for AI Engineers, and subword tokenization is in Tokenization and Context Windows; here we stop at the point where self-attention is motivated, derived and costed.
Sequence interviews are a story with a plot: "recurrence solves variable length, but gradients vanish through time; gates fix that; but a single vector cannot hold a long sentence; attention fixes that; and once you have attention you do not need the recurrence." If you can tell that story with the equations and numbers behind each step, you can answer almost any follow-up. This lesson verifies every number by execution and drills the failure modes (padding in the loss, teacher forcing, batch_first, beam search on open-ended text).
Sequence Problems and Tensor Shapes
A batch of sequences is a tensor of shape (batch, time, features): for token ids it is (B, T) of integers that an embedding layer turns into (B, T, E); for audio it may be (B, T, 80) log-mel frames; for sensor data (B, T, channels). Sequences in a batch have different lengths, so shorter ones are padded to the longest with a pad id (or zeros) and a length vector or boolean mask records which positions are real. Everything downstream (the recurrence, the loss, pooling, attention) must respect that mask; ignoring it is the single most common sequence bug.
| Pattern | Input to output | Example | Typical head |
|---|---|---|---|
| Many-to-one | T steps to one label | sentiment, intent, anomaly flag | last hidden state (or pooled outputs) to a classifier |
| One-to-many | one vector to T steps | image captioning, music generation | decoder initialized from the vector |
| Many-to-many, aligned | T to T | POS tagging, per-frame audio labels, per-step forecasts | an output at every step |
| Many-to-many, unaligned (seq2seq) | T_{in} to T_{out} | translation, summarization, speech recognition | encoder, then autoregressive decoder |
| Language modelling | predict the next element | LLMs, autocomplete | softmax over the vocabulary at every step |
PyTorch trap: nn.RNN, nn.LSTM and nn.GRU default to time-first inputs (T, B, features); pass batch_first=True if your tensors are (B, T, features). Forgetting it does not raise an error. A (4, 10, 16) tensor fed to nn.LSTM(16, 32) is silently treated as 4 time steps of a batch of 10, and the output shape is (4, 10, 32) (verified). The model trains, just on garbage.
Two ways to deal with padding in a recurrent model are covered below under Training Details: packed sequences (the recurrence skips pad steps) and masks (the recurrence runs, but loss and pooling ignore the pad outputs).