Practice — Model Compression, Quantization & On-Device ML (6 questions)

Your team serves a 6.74B-parameter chat model at fp16 on A100-80GB GPUs (about 2.0 TB/s of memory bandwidth) at batch size 1 for a latency-sensitive assistant. A colleague proposes "just quantize it to 4 bits, it will be 4x faster and nobody will notice". You are asked to decide.

1. Compute the weight memory at fp16, int8 and int4 with group size 128 and an fp16 scale per group. What is the real bits per weight at group 128, and why is it not exactly 4?
2. Estimate the batch-1 decode ceiling in tokens per second for each format, from bandwidth over bytes. Then add a single 4,096-token sequence with an fp16 KV cache (32 layers, 32 heads, head dimension 128). How much does the int4 speedup shrink, and what would an int8 KV cache change?
3. Explain why this speedup does not apply to prefill or to large batches, and what kind of quantization (weight-only or weight-and-activation) you would pick for each regime.
4. Write the evaluation plan you would require before shipping the int4 model. Include what you would compare it against besides the fp16 original, and the acceptance gate.

You post-training-quantize a mobile image classifier (depthwise-separable convolutions) to int8 for a phone. Float accuracy is 74.0 percent; the int8 model scores 68.0 percent. The tool used per-tensor scales for weights and activations and a calibration set of 32 images, all taken from a stock-photo dataset, while production images come from phone cameras in poor light.

1. Explain, with a small numeric example, why one scale per tensor can be disastrous for the weights of a depthwise convolution. What do you change?
2. Explain what is wrong with the calibration set and how min-max calibration interacts with outliers. Give the fix.
3. After those fixes the drop is still 1.8 points on the target slice. Describe the straight-through estimator and when you would move to quantization-aware training. What does QAT cost?
4. List the checks you run to make sure the on-device result matches what you evaluated.