Model Compression, Quantization & On-Device ML
Shrinking models for latency, cost and the edge without quietly losing accuracy
Shrink a model for latency, cost or a phone and prove you did not break it: where the time and memory go (weights versus activations, memory-bandwidth-bound decode, energy and thermal limits on devices), quantization fundamentals (int8, int4, fp8, NF4 formats, symmetric versus asymmetric affine quantization with scale and zero-point worked by hand, per-tensor versus per-channel versus per-group granularity with measured error, calibration and clipping, int32 accumulation at the matmul verified), PTQ versus QAT (fake quantization and the straight-through estimator, with a 5-seed experiment showing QAT recovers 2-bit accuracy that PTQ loses), LLM quantization (the outlier-channel problem, SmoothQuant, GPTQ and AWQ ideas reproduced as toys, weight-only versus weight-and-activation, NF4 and GGUF families, KV-cache quantization, evaluation beyond perplexity), pruning and sparsity (unstructured versus structured, why zeros in a dense tensor save nothing, N:M 2:4 sparsity, iterative magnitude pruning measured, lottery tickets), knowledge distillation (soft targets, temperature and the T-squared factor verified by gradient, feature and sequence-level distillation, what actually drives the gain), low-rank factorization and LoRA merging (and why merging into int4 weights silently loses the adapter), compilers and runtimes (ONNX export pitfalls reproduced, TensorRT, torch.compile, operator fusion, BatchNorm folding verified), on-device ML (Core ML, LiteRT, ExecuTorch, NNAPI, constraints, the on-device versus server decision matrix, delivery, versioning and on-device A/B), honest benchmarking (p50 versus p99, warm versus cold, batch 1, device variance, accuracy-latency Pareto), and a worked example sizing a 7B model at fp16, int8 and int4 with decode-speed ceilings and an evaluation plan, plus a distill-then-int8-QAT plan for a phone vision model.
Practice questions (6)
-
View →
Will a 7B Model at int4 Be Cheaper and Fast Enough? Size, Decode Ceiling, Evaluation
Intermediate · Free -
View →
The int8 Model Lost 6 Points: Diagnose Granularity, Calibration and PTQ versus QAT
Intermediate · Free -
View →
W8A8 Collapsed on a 13B LLM: Outliers, SmoothQuant, GPTQ, AWQ and What to Ship
Advanced -
View →
'80 Percent Sparse, 5x Faster' and a Distillation Plan: Evaluate the Claims
Advanced -
View →
ONNX Export Works on Your Laptop and Fails in Production: Export, Fusion and Benchmark Hygiene
Advanced