Model Compression, Quantization & On-Device ML
This subject assumes Distributed Training & GPU Systems (memory-bound versus compute-bound, the roofline, bf16/fp16/fp8 ranges), Transformers for AI Engineers (parameter counts and the KV-cache formula) and The PyTorch Training Loop from Scratch. The boundary: serving-engine internals (continuous batching, PagedAttention, speculative decoding, capacity planning) belong to LLM Inference Engineering; choosing and running open-weight models with Ollama, llama.cpp or vLLM belongs to Local LLM Deployment and Open-Weight Serving; the adapter training recipes belong to Fine-Tuning: SFT, LoRA/QLoRA, RLHF and DPO. Here we cover the compression techniques themselves, how to evaluate that they did not break the model, and the on-device decision.
Interviews ask this as "the model is too slow, too big or too expensive; what do you do?" The senior answer is a sequence, not a trick: measure where the cost is, pick the cheapest technique that attacks it, prove quality with an evaluation that can actually see the damage, and benchmark on the real target. Everything below that could run on a CPU box was executed (PyTorch 2.14 CPU, ONNX Runtime 1.31, tiny models and toy tasks) and is marked "(verified)". Hardware figures (A100/H100 bandwidth and peaks), paper details and library status are published facts quoted from memory, not measured here; treat them as approximate and check current documentation. CPU wall-clock numbers say nothing about GPU or phone behaviour.
Why Compress: Where the Time, Memory and Energy Go
Three different budgets, three different bottlenecks:
| Budget | Typical limit | What dominates | Techniques that attack it |
|---|---|---|---|
| Memory / model size | GPU VRAM, phone RAM, app download size | weights (\Psi \times bytes per parameter), plus KV cache for LLMs | quantization, pruning (structured), distillation, low-rank |
| Latency | p99 SLO, 30 ms camera frame, tokens per second | decode is memory-bandwidth-bound: every generated token reads all weights once, about 1 FLOP per byte (Distributed Training & GPU Systems); CNN inference is often compute-bound | quantization (fewer bytes), distillation (fewer FLOPs), fusion/compilers |
| Cost / energy | dollars per million requests, battery, thermal throttle | bytes moved: a DRAM access costs orders of magnitude more energy than an arithmetic operation (Horowitz 2014, popularized by Han et al. 2015; from memory) | anything that shrinks the bytes moved |
The key insight for LLMs: at batch 1 decode time is roughly weight bytes divided by memory bandwidth, so halving the bytes per weight nearly halves the time per token, with no change to FLOPs. For large batches or prefill the GPU is compute-bound and weight-only quantization buys little (below). On a phone the equivalent is that memory traffic costs battery and heat: a model that is fast for 10 seconds and then throttles after two minutes of sustained use is not fast. Finally, size is a product constraint: app-store and over-the-air download limits, cold-start time to load the model, and update bandwidth all grow with bytes.
graph TD
A["Model too slow / big / costly"] --> B["Profile: weights, KV cache, activations, kernel launch overhead?"]
B -->|"bytes (memory-bound)"| C["Quantize weights (int8 / int4), prune structure, shrink KV cache"]
B -->|"FLOPs (compute-bound)"| D["Distill to smaller model, structured pruning, W8A8 / fp8, N:M sparsity"]
B -->|"overhead / many small ops"| E["Fuse + compile: ONNX Runtime, TensorRT, torch.compile, CUDA graphs"]
C --> F["Evaluate on tasks, not perplexity alone"]
D --> F
E --> F
F --> G["Benchmark on the real target at the real batch size"]