Practice — Transformers for AI Engineers (5 questions)

Your product team wants to double the maximum conversation length your chat product supports, from 16k to 32k tokens of context. Finance pushes back: the infra team's early load test shows GPU memory usage for concurrent sessions went up by far more than 2x, and inter-token latency on long conversations got noticeably worse too. Nobody on the product team understands why "twice the context" isn't "twice the cost."

1. Explain, mechanistically, what is actually consuming the extra memory, and why it scales the way it does.
2. Sketch a rough number: for a 13B-parameter model with 40 layers, d_{model}=5120, GQA with 8 KV heads (head dim 128), fp16 throughout, how much KV cache memory does one 32k-token conversation need? How does that compare to the model weights themselves?
3. Why did per-token latency get worse too, not just memory usage?
4. Name two architectural or serving levers you could pull if you wanted longer context without this scaling hitting you as hard.

Your company is building a document-translation feature: users upload a document (up to ~20k tokens) in one language and get back a full, faithful translation in another. A teammate proposes using your existing decoder-only chat LLM (the one already deployed for the product's chat assistant) with a "translate the following document:" prompt, since "it's already a great LLM and reusing it means no new infra."

1. Is that a reasonable default, or does the task's shape argue for something else? Walk through the reasoning, not just the conclusion.
2. If you did consider an encoder-decoder model for this feature, what would you actually be trading off against the "just prompt the LLM you already have" approach?
3. What would make you say "no, decoder-only prompting is clearly the right call here, don't overthink it"?

Your model is underperforming on a domain-specific extraction task (pulling structured fields out of contracts). Leadership's proposed fix: "swap the 8B model we're using for the 70B version of the same model family — bigger models are better, this should just work."

1. Is parameter count alone a good predictor of whether this fixes the problem? Explain what it does and doesn't tell you.
2. What two or three questions would you actually want answered before recommending for or against the swap?
3. Give a concrete alternative explanation for the underperformance that a bigger model wouldn't fix, and how you'd distinguish it from "the model is just too small for this task."

Your infra team proposes fine-tuning your self-hosted model to use a more aggressive grouped-query attention configuration — going from 16 KV heads down to 4 — specifically to cut KV cache memory and let you raise the concurrent-request limit on your current GPUs.

1. Explain what's actually changing in the attention mechanism, and why it reduces KV cache size the way it does.
2. Does this change affect the query-side computation or only the key/value side? Why does that distinction matter for the trade-off you're evaluating?
3. What would you want to check before signing off on this change, beyond "the cache got smaller"?

Your team fine-tuned a model that was originally trained with an 8k token context window, and now wants to serve it at a 64k context window because the product needs to summarize long documents. Early testing shows the model handles the first ~8k tokens of a long document fine, but coherence and instruction-following degrade noticeably for content further into the document, well before the 64k limit is reached.

1. Why does this happen even though attention, mechanically, is perfectly well-defined at 64k tokens — nothing crashes, nothing is out of bounds?
2. What is the connection between this failure and the model's positional encoding scheme specifically?
3. Name a mitigation you'd propose, and be explicit about whether it requires retraining or can be applied at serving time.