Transformers for AI Engineers

An ML-research interview asks you to derive backpropagation through attention, propose an architecture ablation, or argue about scaling laws from first principles. An AI Engineer interview asks a different question: you are going to call these models through an API, choose which one to use, pay for every token, and hit a wall (a context limit, a latency budget, a cost line item) that only makes sense if you understand what is happening inside the box. Nobody expects you to implement a transformer from scratch on a whiteboard. They expect you to explain why the model behaves the way it does in production — why doubling context length doesn't just cost more, it costs disproportionately more in a specific place; why a "70B model" is not simply "10x a 7B model" in the way that matters for your use case; why serving cost per token drops after the prompt but not to zero.

A weak answer in this space sounds like a glossary entry: "attention lets tokens look at other tokens," "the KV cache speeds things up," "decoder-only models are what GPT uses." All true, none of it load-bearing. A strong answer connects mechanism to consequence: "attention is a weighted average over value vectors, weighted by how much a query and key agree, scaled by 1/\sqrt{d_k} so the softmax doesn't saturate as head dimension grows — and because that averaging is symmetric in position, the model needs an explicit positional signal or 'the cat sat on the mat' and 'the mat sat on the cat' are indistinguishable." The strong answer can also do arithmetic: given a model size, a context length and a batch size, sketch how many gigabytes of KV cache that context is going to cost you, and say what lever you'd pull if it doesn't fit.

This subject is the opening piece of the AI Engineer Interview track, and it's deliberately concept-heavy rather than a worked system design — think of it as the vocabulary and mental models that the rest of the track (RAG, agents, evaluation, serving) leans on without re-deriving. Wherever a topic here determines a real production cost or constraint, it says so, and it points to model-serving-and-deployment and model-training-and-experimentation for the subjects that go deep on those production concerns.

Scaled Dot-Product Attention

At the core of every transformer layer is one operation, applied over and over:

In words: for every query vector, compute how similar it is to every key vector (a dot product), turn those similarities into a probability distribution (softmax), and use that distribution to take a weighted average of the value vectors. The output for a given position is a blend of the values at every position it attends to, weighted by relevance. That's it — attention is a differentiable, content-based lookup table.

Why it works. Unlike a fixed-window convolution or a recurrent hidden state, attention lets every token gather information directly from every other token in one step, regardless of distance. There's no vanishing-gradient chain to walk back through 500 timesteps (as in an RNN) and no receptive-field limit to stack layers around (as in a CNN). The trade-off, and it's the single most consequential trade-off in the whole architecture, is that computing QK^\top is O(n^2) in sequence length n — every token compares itself to every other token. That quadratic term is the reason context length is expensive, and it's the reason later sections of this subject (the KV cache, the worked examples) matter as much as they do.

Why the \sqrt{d_k} scaling. If Q and K have components drawn independently with roughly unit variance, the dot product of two d_k-dimensional vectors has variance that grows linearly with d_k — the raw scores get large as dimensionality increases. Large scores pushed through softmax saturate: one score dominates, the gradient with respect to the others goes to nearly zero, and training destabilizes. Dividing by \sqrt{d_k} keeps the variance of the pre-softmax scores roughly constant regardless of head dimension, so the softmax stays in a well-conditioned regime. This is a detail worth having ready — it's the kind of thing that separates "I've read the paper" from "I understand why the paper does what it does."

Self-attention vs. cross-attention. These are the same mechanism with different sources for Q, K and V:

• Self-attention: Q, K and V are all derived from the same sequence. Every token attends to (a subset of) the other tokens in its own sequence. In an encoder, that attention is bidirectional — a token can attend to tokens before and after it. In a decoder, it's causal — a masked variant where token t can only attend to tokens 1..t, which is what makes autoregressive generation well-defined (the model can't peek at the answer it hasn't generated yet).
• Cross-attention: Q comes from one sequence (the decoder's current state), while K and V come from a different sequence (the encoder's output). This is how an encoder-decoder model lets the thing it's generating look back at the thing it was given — a translation decoder attending to the source sentence, for instance.

Decoder-only models, which is most of what you'll work with day to day, don't have cross-attention at all — there's no separate encoded sequence to attend to. Instead, the prompt and the generated continuation live in one token stream, and "attending back to the prompt" is just ordinary causal self-attention over earlier positions in that same stream.

Multi-Head Attention

Running one attention operation over the full model dimension d_{model} works, but it forces every token pair to be summarized by a single similarity score and a single blended output. Multi-head attention splits Q, K and V into h smaller subspaces (each of dimension d_k = d_{model}/h), runs scaled dot-product attention independently in each subspace, then concatenates the results and applies one output projection:

input (d_model)
   │
   ├─ head 1: project to d_k, attention, output (d_k)  ──┐
   ├─ head 2: project to d_k, attention, output (d_k)  ──┤
   ├─ ...                                                ├──► concat (d_model) ──► output projection ──► d_model
   └─ head h: project to d_k, attention, output (d_k)  ──┘

The compute cost is roughly the same as one full-size attention operation (you've just partitioned the same total dimensionality into h pieces), but the model gets h independent "views" it can specialize — one head might learn to track subject-verb agreement, another might learn positional adjacency, another might track coreference. Empirically, different heads do specialize this way, though not always in interpretable directions. The interview-relevant point isn't the interpretability story; it's that multi-head attention buys you representational diversity at essentially the same FLOP budget as single-head attention at the same total width.

One production-relevant variant worth naming: grouped-query attention (GQA) and its extreme, multi-query attention (MQA). Both reduce the number of distinct K/V projections (sharing them across multiple query heads) while keeping the number of query heads unchanged. This doesn't change the attention mechanism conceptually — it changes how much you have to store per token, which is exactly the KV cache problem the next section covers. Nearly every recent production LLM (Llama 3, Mistral, and others) uses GQA specifically because of its effect on cache size, not because it improves quality.

Decoder-Only vs. Encoder-Decoder Architectures

The original Transformer (2017) was encoder-decoder, built for machine translation. Almost every LLM you'll integrate against today — GPT-4, Claude, Llama — is decoder-only. Knowing why that shift happened, and where the older shape still earns its keep, is a standard interview beat.

Encoder-decoder (T5-style):                    Decoder-only (GPT-style):

 source tokens                                  prompt tokens + generated tokens
      │                                                     │
      ▼                                          ┌──────────▼──────────┐
 ┌────────────┐  bidirectional self-attn          │  causal self-attn    │  token t attends
 │  encoder   │  (sees whole source at once)       │  (token t attends    │  only to 1..t
 │  N layers  │                                    │   only to 1..t)      │
 └─────┬──────┘                                    │       ×N layers      │
       │ encoder output (K,V)                      └──────────┬──────────┘
       ▼                                                       ▼
 ┌────────────┐  causal self-attn over output-so-far   next-token prediction,
 │  decoder   │  + cross-attn into encoder output      one stream, one objective
 │  N layers  │  (Q from decoder, K/V from encoder)
 └─────┬──────┘
       ▼
 next-token prediction (target sequence)

Why decoder-only won for general-purpose LLMs:

1. One objective, one stack, cleaner scaling. Next-token prediction over a single stream is simple to implement, simple to parallelize during training (every position is supervised in one forward pass via teacher forcing), and the scaling-law literature that justified training ever-larger models was built almost entirely around this objective. Simplicity at scale wins.
2. Input and output are the same kind of thing. A decoder-only model doesn't need a task-specific split between "the encoder's job" and "the decoder's job" — instructions, context, retrieved documents, tool schemas and the model's own output all live in one token stream. That uniformity is exactly what makes prompting, few-shot examples, chat turns and tool use all expressible as "more tokens in the stream" instead of requiring architectural changes per task.
3. It composes with the KV cache. Because generation is just causal self-attention continuing over the same stream, the caching mechanism described below applies uniformly to the prompt and the generated continuation. An encoder-decoder model needs to cache both the encoder's output and the decoder's growing self-attention cache — decoder-only collapses that into one mechanism.
4. Emergent in-context learning. The GPT-3-era finding that a large enough decoder-only model can perform new tasks from a few examples in the prompt — with no gradient update — is tied to this "everything is one stream, predict the next token" framing, and it's the capability that made prompting (rather than fine-tuning per task) the default way to use LLMs.

Where encoder-decoder still matters: when the task has a genuinely asymmetric shape — a long, information-dense input that benefits from full bidirectional context, and a comparatively short, well-scoped output — translation being the canonical case. It's also worth knowing that encoder-only models (BERT and its descendants) haven't disappeared either: they're not generative at all, but they're still the standard choice for embedding models, rerankers and classifiers, which is to say the retrieval half of a RAG pipeline rather than the generation half. If an interviewer asks "would you ever reach for something other than a decoder-only LLM," that's the honest answer: encoder-only for representations, encoder-decoder for a handful of translation-shaped tasks, decoder-only for everything else.

Why the KV Cache Exists

This is the mechanism most AI Engineer interviews actually want you to be able to reconstruct, because it's the reason autoregressive generation is usable at all rather than a theoretical curiosity.

The problem it solves. Generating text one token at a time is inherently sequential — token t+1 depends on token t. Naively, to generate token t+1, you'd re-run the entire forward pass over tokens 1..t: every layer's query/key/value projections, every attention computation, every MLP, for every one of the t tokens, from scratch. Generating an n-token response this way costs \sum_{t=1}^{n} O(t) total token-processing work — quadratic in n, and dominated by pure waste, because tokens 1..t-1 haven't changed since the last step; you'd be recomputing identical numbers over and over.

What gets cached. The key insight is that the K and V projections for a given token, at a given layer, never change once computed — they depend only on that token's own hidden state at that layer, not on what gets generated afterward. So the harness stores them: after processing token t, keep its K and V vectors, for every layer, in a cache. At step t+1, you only run the forward pass for the new token — project its Q, K, V; compute attention using its Q against the cached K/V for tokens 1..t plus its own; run its MLP; append its new K/V to the cache. The per-step work becomes proportional to the current context length only for the attention step (the query still has to compare against every cached key), not for re-deriving the earlier tokens' representations. Total generation cost drops from quadratic re-derivation to linear-in-context-length incremental work — which is the difference between "usable product" and "not viable at any context length worth having."

Without KV cache:                       With KV cache:
step 1: process [t1]                    step 1: process [t1] → cache K1,V1
step 2: process [t1,t2]  (redo t1!)     step 2: process [t2] only, attend to cached K1,V1 → cache K2,V2
step 3: process [t1,t2,t3] (redo t1,t2!)step 3: process [t3] only, attend to cached K1..K3,V1..V3 → cache K3,V3
...                                     ...
step n: process [t1..tn]  (redo all!)   step n: process [tn] only, attend to cached K1..Kn-1,V1..Vn-1
total: O(n²) redundant work             total: O(n) new-token work + O(n) cache reads

Why the cache grows the way it does, and why that's a serving problem. The cache stores one K and one V vector per token, per layer, per (KV) head. Its size is:

where the leading 2 is for K and V, L is the number of layers, H_{kv} is the number of KV heads (which is why GQA/MQA — fewer KV heads — directly shrinks this), and d_{head} is the per-head dimension. Every one of those factors is linear, so the cache grows linearly with context length and linearly with batch size — and it has to live in the same GPU memory as the model weights. This is the direct link between an architectural detail and a line item on a cloud bill: a longer context window or more concurrent requests doesn't just mean "more tokens to process," it means a proportionally larger chunk of expensive GPU memory reserved for cache instead of for serving more requests. model-serving-and-deployment covers the resulting serving techniques (continuous batching, paged KV cache allocation, quantizing the cache itself) in depth; this subject is about why the constraint exists in the first place.

What Parameter Count Does and Doesn't Tell You

"How big is the model" is often treated as a proxy for "how good is the model," and in an AI Engineer interview, pushing back on that with the right nuance is a signal of production maturity, not pedantry.

Parameters vs. FLOPs vs. data. Parameter count N tells you the model's capacity — roughly, how much information it could store — but says nothing on its own about how much it was trained on or how well that capacity was used. Training compute is approximately C \approx 6ND FLOPs, where D is the number of training tokens (the factor of 6 comes from two FLOPs per parameter per token for the forward pass, and roughly four more for the backward pass). Two models with the same N can differ enormously in quality depending on D and on data quality — and the reverse holds too: the Chinchilla scaling-law finding (2022) was precisely that many earlier large models were undertrained relative to their size — a lot of the era's "bigger is better" models would have performed better as a smaller model trained on proportionally more tokens for the same total compute budget. The interview-ready version of this: parameter count is one input to a joint optimization over parameters, data and compute, not a standalone quality metric. "Bigger model" is the right lever specifically when you're compute-constrained on inference and want more capacity per forward pass — it is not automatically the right lever when the actual bottleneck is data quality, task fit, or how well the model was trained for the compute it received. model-training-and-experimentation is where the training-infrastructure side of this — how you'd actually spend a training compute budget — is covered in depth.

Where the parameters live. A rough mental model, useful for reasoning about which lever moves cost and which moves capability:

Per transformer layer, the MLP block has roughly twice the parameters of the attention block — for a hidden size d_{model}=4096 (a small-to-mid-size model), attention is about 4 \times 4096^2 \approx 67M parameters per layer, while the MLP is about 8 \times 4096^2 \approx 134M. Stack that over dozens of layers and the MLP blocks dominate the parameter budget of any model past a few billion parameters — which is exactly why most of the recent efficiency techniques aimed at parameter count (mixture-of-experts sparsity, in particular) target the MLP blocks specifically, not attention. Attention gets the architectural spotlight because of its quadratic compute story; the MLP quietly holds most of the weights.

Positional Encoding

Self-attention, as defined above, is a weighted average over values — and averaging is a set operation. If you permuted the order of the input tokens (and permuted the positions the model reports them at consistently), the raw attention computation would produce the same set of outputs, just relabeled — the mechanism itself has no notion of "before" or "after." Without an explicit signal, "the cat sat on the mat" and "the mat sat on the cat" would be indistinguishable to the attention operation. Positional encoding is how the model is told what order the tokens came in.

Absolute encodings. The original Transformer added a fixed sinusoidal function of position to each token's embedding before the first layer; later models (early GPT/BERT) used a learned embedding per absolute position instead. Both work, but both bake in a hard assumption: the model only ever sees positions up to whatever it was trained on, and behavior at unseen positions (a longer sequence than anything in training) is undefined — the model has no principled way to generalize past its trained context length.

Rotary position embedding (RoPE), conceptually. Rather than adding a positional vector, RoPE rotates the query and key vectors by an angle that depends on their absolute position, in a way carefully constructed so that the dot product Q_i \cdot K_j after rotation depends only on the relative distance i - j, not on the absolute positions i and j themselves. That relative-position property is the useful part: two tokens that are 5 apart interact the same way whether they're at positions (10, 15) or (10,000, 10,005), which is a much more natural inductive bias for language than "absolute position 10,003 means something specific." RoPE is now close to universal in decoder-only LLMs (Llama, GPT-NeoX-style models, and most current open and closed models) for exactly this reason, and it composes cleanly with the KV cache — rotation is a deterministic function of a token's own position, so a cached key doesn't need to be recomputed when later tokens are added.

Context-length extrapolation. RoPE's relative-distance property is what makes it possible to extend a model's context window after the fact rather than only ever using what it was trained on — but it isn't unlimited: the rotation is periodic, tied to a base frequency, and relative distances the model never saw during training (very long ones) still produce degraded behavior even though the mechanism is mathematically defined for them. Techniques like position interpolation and NTK-aware/YaRN-style frequency rescaling adjust RoPE's base frequency so that long-context behavior at inference time maps back onto distances the model effectively did see relative representations for during training, letting a model trained at, say, an 8k context be usably extended to 32k or beyond without full retraining. The interview-relevant takeaway: "just increase the context window" is a claim about the positional encoding scheme as much as it is about attention compute and KV cache memory — all three constraints move together.

Worked Example: Sizing a KV Cache

Take a 70B-parameter-class decoder-only model as a concrete case, with illustrative but realistic architecture numbers: 80 layers, d_{model}=8192, 64 query heads (head dim 128), and — the detail that matters most here — grouped-query attention with only 8 KV heads (a 8:1 grouping ratio, similar in spirit to production models at this scale). Weights stored in fp16 (2 bytes/parameter), KV cache also stored in fp16 (2 bytes/value).

Per-token cache cost, using the formula from the KV cache section:

Scale that across context lengths and a modest batch of concurrent requests:

Compare that to the model weights themselves: 70B parameters at fp16 is ~140 GB, loaded once. At a 128k context with just 8 concurrent long-context requests, the KV cache alone would need more memory than the model weights — on real hardware, that's the difference between "fits on one node" and "needs cache eviction, quantization, or a different serving strategy entirely," which is precisely why GQA (an 8:1 ratio here, versus a hypothetical 64:1 with full multi-head attention) exists: run the same numbers with 64 KV heads instead of 8 and the 128k-context, batch-of-8 figure becomes roughly 2.6 TB — not a design choice you can serve your way out of.

Prefill vs. decode cost per token. These two phases of generation have opposite bottlenecks, and conflating them is a common interview slip:

• Prefill (processing the prompt) runs the entire prompt through the model in one parallel forward pass. Rough FLOPs are \approx 2 \times N_{params} \times n_{tokens}. For a 7B model and a 2,000-token prompt, that's roughly 2 \times 7\text{e}9 \times 2000 \approx 2.8\text{e}13 FLOPs — on a GPU delivering a few hundred TFLOPS of effective throughput, that's on the order of tens to low hundreds of milliseconds. Prefill is compute-bound: the GPU's matrix units are kept busy, and throughput scales with how parallel the hardware is.
• Decode (generating each subsequent token) processes exactly one new token per step, so the FLOPs per step are tiny — for the same 7B model, roughly 2 \times 7\text{e}9 \times 1 \approx 1.4\text{e}10 FLOPs, a few milliseconds of compute at most. But each step still has to read the entire model's weights (and the growing KV cache) from GPU memory to process that one token: at 14 GB of fp16 weights and a few TB/s of memory bandwidth, that's several milliseconds just to move the weights, before counting the KV cache read — and that memory-read cost barely shrinks per-token no matter how little compute the step needs. Decode is memory-bandwidth-bound, not compute-bound, which is the whole reason batching multiple requests together for decode is such a large serving win (the same weight read is amortized across many tokens at once) and why a longer context makes every subsequent decode step slightly slower (more KV cache to read, not more FLOPs to run). model-serving-and-deployment builds directly on this distinction — continuous batching exists specifically to fix decode's memory-bandwidth problem.

The one-line summary worth having ready: prefill is fast because it's parallel and compute-bound; decode is comparatively slow per token because it's sequential and memory-bandwidth-bound — and everything about serving cost (batching, caching, quantization) is aimed at that second problem.

Why batching decode requests together is such a large win. Reading 14 GB of weights from HBM to process a single token is nearly the same amount of memory traffic as reading those same 14 GB to process 32 tokens' worth of decode work at once (one weight read serves every request in the batch, since they all multiply against the same weight matrices). So batching 32 concurrent decode requests together doesn't cost anywhere near 32x the memory traffic — it's closer to 1x traffic for roughly 32x the useful output, until the batch gets large enough that compute (which does scale with batch size) catches up to and then exceeds the memory-bandwidth cost. That crossover point — where a decode step goes from memory-bound to compute-bound as batch size grows — is exactly the number a serving team tunes towards, and it's the mechanistic reason a production LLM API can serve many users' conversations concurrently on the same GPUs at a small fraction of the marginal cost of one.

Follow-Up Questions Interviewers Ask

• "Why not just make attention non-quadratic and skip all this KV cache complexity?" — People have tried (linear attention, state-space models like Mamba, sliding-window attention). Each trades away something: linear-attention approximations generally lose some of the precise long-range recall that full attention gives you, and most production-grade LLMs today still use standard quadratic attention plus a KV cache rather than a fundamentally sub-quadratic mechanism, precisely because the quality trade-off hasn't clearly been worth it yet at the frontier. Sliding-window attention (bound attention to a fixed recent window) is used in some production models as a partial mitigation, at the cost of not attending to arbitrarily distant tokens at all.
• "If MLP blocks hold most of the parameters, why does everyone talk about attention?" — Attention is architecturally the more novel, more discussed piece (it's the idea the "Attention Is All You Need" paper is named for, and it's what gives the quadratic-cost story that shapes serving economics), but "novel and discussed" isn't the same as "holds the most weights." Interviewers who ask this are checking whether you conflate architectural significance with parameter share.
• "Does a bigger KV cache mean better quality?" — No — cache size is a serving-cost variable, not a quality variable. It's determined by architecture (layers, KV heads, head dim) and how much context you're holding, not by how good the model's answers are. Don't let cache-size questions bleed into quality claims.
• "Why does Claude/GPT-4 not tell you its exact parameter count?" — Increasingly, frontier labs don't publish exact figures, partly competitive and partly because — per this whole subject — parameter count alone is a poor single proxy for what people actually want to know (capability, cost per token, latency). The honest engineering answer to "how big is it" is usually "I don't know, and for my purposes I care about its measured latency, cost per token and eval scores instead."
• "Could you extend context length just by increasing the KV cache budget, with no other changes?" — No — that only addresses the memory-to-hold-it problem. The model also needs a positional encoding scheme that behaves sensibly at those lengths (see the RoPE/extrapolation discussion) and, ideally, some exposure during training or fine-tuning to sequences near that length; a bigger cache budget alone doesn't fix degraded attention behavior at unseen relative distances.

Common Mistakes and Interview Traps

• Reciting the attention formula without explaining \sqrt{d_k}. If you write QK^\top/\sqrt{d_k} on a whiteboard and can't say why the scaling term is there, the interviewer will ask, and "it's in the paper" is not an answer.
• Saying "decoder-only is better" without saying why, or without knowing when it isn't the right answer. The honest framing is "decoder-only won for general-purpose LLMs because of X, Y, Z" — not "encoder-decoder is obsolete." Encoder-only embedding/reranker models are still everywhere in production RAG systems.
• Treating the KV cache as a minor implementation detail. It's the mechanism that makes autoregressive generation linear instead of quadratic, and its memory footprint is a direct, computable production cost — an interviewer asking about context-window pricing or concurrent-request limits is often really asking about KV cache sizing.
• Confusing parameter count with model quality. "It's a 70B model so it's better" ignores training data volume, data quality, and whether the model was trained to compute-optimal proportions in the first place.
• Not knowing where the parameters actually are. Assuming attention dominates a model's parameter budget (it doesn't — MLP blocks typically do, by roughly 2:1) leads to reasoning backwards about which efficiency techniques (MoE sparsity vs. attention variants like GQA) target which part of the model and why.
• Treating positional encoding as a footnote. It's the answer to "why can't you just feed in an arbitrarily long context," and RoPE's relative-position property is directly why context extension techniques exist at all.
• Confusing prefill and decode. Quoting a single "tokens per second" number without distinguishing the compute-bound prefill phase from the memory-bandwidth-bound decode phase misses why time-to-first-token and inter-token latency behave so differently, and why batching helps one far more than the other.
• No numbers. An AI Engineer interview rewards being able to sketch an order-of-magnitude estimate (KV cache GB, decode latency, cost per token) live, with stated assumptions — not memorized figures. State the assumption, do the arithmetic, move on.

Key Takeaways

• Attention is a differentiable, content-based weighted average: \text{softmax}(QK^\top/\sqrt{d_k})V. The \sqrt{d_k} scaling keeps the softmax well-conditioned as head dimension grows; the O(n^2) cost in sequence length is the root cause of nearly every context-length-related production constraint you'll encounter.
• Multi-head attention runs several smaller attention operations in parallel subspaces at roughly the cost of one full-size operation, buying representational diversity; GQA/MQA reduce the number of distinct KV heads specifically to shrink the KV cache, not to change what attention computes.
• Self-attention draws Q, K, V from the same sequence (causal in decoders, bidirectional in encoders); cross-attention lets a decoder attend into a separate encoded sequence. Decoder-only models have no cross-attention — the prompt and the generation share one causal stream.
• Nearly every modern LLM is decoder-only because one objective (next-token prediction) over one unified input/output stream scales cleanly, composes naturally with the KV cache, and underlies in-context learning; encoder-decoder still fits strongly asymmetric input/output tasks like translation, and encoder-only models remain the default for embeddings and rerankers.
• The KV cache exists because recomputing every prior token's key/value projections at every generation step would make autoregressive decoding quadratic in redundant work; caching them makes each new token's cost proportional to context length rather than to the whole regenerated history. Cache size scales linearly with layers, KV heads, head dimension, context length and batch size — and it competes with model weights for the same GPU memory, which is a direct line to serving cost (see model-serving-and-deployment).
• Parameter count is one input to a joint optimization with training data volume and compute (Chinchilla's core lesson); "bigger model" is not automatically the right lever. Within a model, MLP blocks typically hold roughly twice the parameters attention blocks do per layer, which is why parameter-efficiency techniques (MoE) target MLPs specifically.
• Positional encoding exists because attention is otherwise permutation-invariant. RoPE encodes relative position by rotating Q/K vectors, which is what makes context-length extension techniques (position interpolation, NTK/YaRN scaling) possible without full retraining.
• Prefill is compute-bound and parallel (fast per token); decode is memory-bandwidth-bound and sequential (slow per token, and slower still as the KV cache grows) — nearly every serving optimization in production (model-serving-and-deployment) exists to address that second constraint.