LLM Inference Engineering

The boundary. Cost and Latency Engineering for LLM Apps is the application level: shorter prompts, caching responses, routing to cheaper models. Local LLM Deployment and Open-Weight Serving is choosing and running an open model: which model, which quantization, how much VRAM, which runtime on which box. This lesson is the serving engine internals underneath both: what a request does on the GPU, where the memory goes, how the scheduler batches, and how you size a fleet from first principles. It assumes the architecture from Transformers for AI Engineers and borrows the memory-bandwidth arguments from Distributed Training & GPU Systems and Model Compression, Quantization & On-Device ML; it does not repeat them.

Interviews for AI engineer and ML system design roles ask the same thing in different clothes: "this model is too slow or too expensive to serve; walk me through it." The answer is arithmetic first: bytes per token of KV cache, bytes moved per decode step, FLOPs per prompt token, and only then an engine feature. Everything that could run on a CPU box was executed with fixed seeds and the output pasted (marked verified). Everything that depends on a real GPU or a real engine (vLLM, SGLang, TensorRT-LLM, FlashAttention kernels, expert parallelism) is explained conceptually and marked NOT MEASURED. Hardware numbers are from memory and approximate (H100 SXM 80 GB: about 3.35 TB/s HBM and 989 TFLOPs bf16 dense; A100 80 GB: about 2.0 TB/s and 312 TFLOPs), the same ones used in the sibling lessons. Timing models are labeled ASSUMPTION: they are back-of-envelope, not benchmarks. Treat every paper and engine claim as from memory, and never quote a throughput number for a real engine that you did not measure on your own traffic.