LLM Inference Engineering

Prefill vs decode, KV cache math, continuous batching, speculative decoding and capacity planning

Reason about how an LLM serving engine works and size one, and defend each choice in an interview: the anatomy of a request (compute-bound prefill versus bandwidth-bound decode, TTFT, TPOT, throughput and goodput, the bytes-per-token latency floor), the KV cache (per-token size formula, a verified cached-versus-recompute toy, the effect of GQA and MQA, PagedAttention fragmentation, prefix caching and KV quantization), batching (static versus continuous in a simulator, chunked prefill and decode stalls, preemption), speculative decoding (acceptance-rate math checked by Monte Carlo, lossless rejection sampling verified empirically, break-even versus batch size), parallelism for serving (tensor, pipeline, expert, prefill/decode disaggregation, multi-LoRA), kernels (FlashAttention, FlashDecoding, CUDA graphs), features that cost latency, the engine landscape, capacity planning and autoscaling signals, observability, and a full worked example for a 70B GQA model at 8k context with 64 concurrent requests. Timing parameters are labeled assumptions; real engines and GPUs were not run.

Practice questions (6)

• How Many 32k-Context Chats Fit on One 80 GB GPU? KV Cache Sizing for an 8B Model

  Intermediate · Free
  View →

• Choose TP and PP to Serve a 405B Dense Model: One Node in fp8 or Two Nodes in fp16

  Advanced · Free
  View →

• Speculative Decoding Break-Even: Which Acceptance Rate and Draft Length Pay Off?

  Advanced
  View →

• Design Autoscaling for an LLM Service: Signals, Thresholds and Cold Starts

  Advanced
  View →

• Capacity Plan for a Traffic Profile: GPUs for 40 Requests per Second

  Advanced
  View →

Quizzes (1)