The int8 Model Lost 6 Points: Diagnose Granularity, Calibration and PTQ versus QAT

1. Per-tensor scales. Depthwise filters have one filter per channel and their ranges can differ by orders of magnitude. Example: two channels with absolute maxima 2.0 and 0.02. A single int8 scale is 2.0/127 = 0.0157; every weight in the second channel is below 0.02, so the channel is represented by at most about 1 to 2 distinct nonzero levels, and a value of 0.01 becomes either 0 or 0.0157 (57 percent off). A per-channel scale for the second channel is 0.02/127 = 0.000157, 100x finer. The same effect, on a three-row example, gave a 6.6x lower mean squared error with per-channel scales, and on a 1,024 x 1,024 matrix with log-normal row scales 35x lower at int8 and 18.5x lower at int4 (verified). Fix: per-channel (per output channel) symmetric weight quantization, which is also what hardware runtimes expect; fold BatchNorm into the convolution before quantizing (otherwise the folded ranges are mis-estimated), and consider equalizing channel ranges for depthwise layers.

2. Calibration. Static activation scales are computed from the activation ranges seen on the calibration data. Thirty-two stock-photo images are too few and from the wrong distribution: night or low-light phone frames have different activation statistics, and values outside the calibrated range are clipped. A toy run calibrated on standard deviation 1 and tested on standard deviation 3 clipped 14 percent of the values and raised the int8 error hundreds of times (verified, toy). Min-max calibration is also fragile in the other direction: one outlier sets the range and the typical values lose resolution, which hurts most at low bit widths (for heavy-tailed data at 4 bits an optimal clip cut the error 5.6x, verified). Fix: a few hundred representative phone images, including difficult lighting, and percentile or MSE-based range observers for activations (asymmetric where the tensor is one-sided, such as after ReLU).

3. STE and QAT. Rounding has zero gradient almost everywhere (verified: the gradient of round is 0). QAT inserts fake-quantization (quantize then dequantize) in the forward pass so the network sees rounding error, and in the backward pass the straight-through estimator treats rounding as the identity, passing the gradient where the value was inside the clipping range and zeroing it where it was clipped. Move to QAT when better PTQ (per-channel, calibration, clipping, cross-layer equalization) leaves a gap above your budget or the bit width is very low; depthwise networks are the classic case. In a toy 2-bit experiment PTQ reached 42.5 percent accuracy and QAT 67.1 percent against 71.9 percent in fp32; at 4 bits both were within one point (verified, 5 seeds). Cost: a fine-tuning run of 10 to 20 percent of the original schedule, access to training data and recipe, and the engineering to make the simulated quantizers match the runtime's real numerics.

4. Parity checks. Compare the exported on-device model's outputs to the PyTorch reference on 1,000 real inputs (cosine similarity and top-1 agreement); evaluate accuracy with the runtime's quantized model, not the simulated one; confirm the whole graph runs on the intended accelerator rather than silently falling back to CPU; test on several device tiers; and keep a golden-input regression test in CI.