Will a 7B Model at int4 Be Cheaper and Fast Enough? Size, Decode Ceiling, Evaluation
Your team serves a 6.74B-parameter chat model at fp16 on A100-80GB GPUs (about 2.0 TB/s of memory bandwidth) at batch size 1 for a latency-sensitive assistant. A colleague proposes "just quantize it to 4 bits, it will be 4x faster and nobody will notice". You are asked to decide.
- Compute the weight memory at fp16, int8 and int4 with group size 128 and an fp16 scale per group. What is the real bits per weight at group 128, and why is it not exactly 4?
- Estimate the batch-1 decode ceiling in tokens per second for each format, from bandwidth over bytes. Then add a single 4,096-token sequence with an fp16 KV cache (32 layers, 32 heads, head dimension 128). How much does the int4 speedup shrink, and what would an int8 KV cache change?
- Explain why this speedup does not apply to prefill or to large batches, and what kind of quantization (weight-only or weight-and-activation) you would pick for each regime.
- Write the evaluation plan you would require before shipping the int4 model. Include what you would compare it against besides the fp16 original, and the acceptance gate.
1. Sizes. At 2 bytes per parameter fp16 is 6.74 \times 2 = 13.48 GB, int8 is 6.74 GB, and int4 with group-128 fp16 scales is 4 + 16/128 = 4.125 bits per weight, so 6.74 \times 4.125 / 8 = 3.48 GB (verified arithmetic). It is not exactly 4 bits because every group of 128 weights also stores a 16-bit scale (an asymmetric scheme with a 16-bit zero-point adds another 0.125 bit: 4.25). Embeddings and the output head are often kept at higher precision, adding roughly half a gigabyte.
2. Decode ceiling. At batch 1 each token reads every weight once, so tokens per second is at most bandwidth over bytes. Using the sizes 13.48, 6.74 and 3.48 GB on 2.0 TB/s: about 148, 297 and 575 tokens per second, an int4 speedup of 13.48/3.48 = 3.9x on weights alone (verified arithmetic; bandwidth is a published spec quoted from memory). The KV cache is read every step too: 2 \times 32 \times 32 \times 128 \times 2 bytes is 524,288 bytes per token, 2.15 GB for 4,096 tokens. The ceilings become 128, 225 and 356 tokens per second, so the int4 speedup over fp16 shrinks from 3.9x to 2.8x. An int8 KV cache halves that term (1.07 GB) and lifts the int4 ceiling to about 440. Real kernels reach only a fraction of the ceiling (dequantization, launch overhead), so the honest claim is "about 2 to 3.5x at batch 1", not "4x", and it must be measured.
3. Regimes. Decode at batch 1 has an arithmetic intensity of about 1 FLOP per byte, far below the A100 ridge of about 156, so it is memory-bound and fewer weight bytes help. Prefill and large batches reuse each loaded weight many times: the GPU becomes compute-bound (fp16 weights cross the ridge near batch 156, int4 weights near batch 39 on this arithmetic). There weight-only int4 still pays a dequantization cost and gives no arithmetic speedup. For the memory-bound decode path choose weight-only (W4A16 with GPTQ or AWQ-style weights); for compute-bound prefill or high-throughput batch serving prefer weight-and-activation int8 or fp8 (SmoothQuant-style W8A8), which speeds up the matmul itself. The serving-engine side is in LLM Inference Engineering.
4. Evaluation plan. (a) Three to five task evaluations that mirror production plus a held-out slice of real prompts, identical decoding settings and prompts for both models, paired comparison with bootstrap confidence intervals (small differences are noise). (b) Long-context retrieval at 4k, 16k and the maximum length, because errors accumulate and the KV cache may be quantized too. (c) JSON or tool-call validity rate, instruction following, multilingual slices (calibration data is often English only). (d) Safety and refusal regressions on the red-team set. (e) Comparisons against int8 and against a smaller fp16 model of the same byte budget, because the right question is whether this is the best use of the bytes. (f) Latency: time to first token, per-token latency and p99 at the real concurrency. (g) A canary with monitoring before full rollout. Acceptance gate fixed in advance, for example no more than a one-point loss on the primary metric and no statistically visible loss on any critical slice. Perplexity alone is not accepted: it is smooth and can look unchanged while structured output or long context degrades.
Share this question