Model Quantization for Edge Inference: Trading a Little Accuracy for Speed

Smaller numbers, faster maths

A trained model's weights and activations are usually 32-bit floats. Model quantization for edge inference converts them to lower precision — 16-bit float or 8-bit integer — so the model is smaller and the arithmetic is faster, especially on hardware built for it.

Why it matters on the edge

A low-power accelerator has limited memory and compute. Some — like an Edge TPU — require int8 to run at all. And the speedup from quantization on the right hardware is large, not marginal. If your detector doesn't fit or doesn't keep pace at full precision, quantization is often what closes the gap.

The two common targets

fp16 (half precision). Near-lossless for most detection models — a straightforward win wherever the hardware supports it. Reported speedups sit in roughly the 2x to 3x range on some workloads.

int8 (8-bit integer). The bigger speedup and the smaller model, but it needs calibration: you run a representative sample of images through the model to determine the right scaling factors, and there's a small accuracy cost you have to measure.

Calibration in practice

Use real shelf photos from varied stores — the same kind of data as your evaluation set — not synthetic images. A bad calibration set gives you a model that's fast and wrong. The calibration set is a real artefact you maintain and refresh as packaging changes.

Measure the accuracy hit

Run the quantized model against your evaluation set and compare detection precision and recall to the full-precision model. On our computer vision project the model's on-device behaviour was something to benchmark directly during the build rather than assume from an adjacent model family — and the same applies to the quantization step. For shelf gap detection the hit is usually small, but "usually" isn't "always."

Quantization-aware training

If the post-training int8 hit is too big, you can train with quantization simulated in the loop. More work, better result — reach for it only when post-training quantization loses too much.

int8 on an Edge TPU, concretely

Some accelerators — the Edge TPU among them — don't run float models at all, so int8 isn't an option, it's the requirement. The pipeline becomes: train in float, export to ONNX, quantize to int8 with a calibration set of real store photos, compile for the accelerator, and validate the compiled model's precision and recall against the float baseline on your evaluation set. A drop of more than two or three points in recall on gap detection is a signal — usually that the calibration set isn't representative of your real conditions, sometimes that the model needs quantization-aware training.

fp16 as the safe middle

Where the hardware supports float16, it's the low-risk default. You get most of the speed and size benefit, almost none of the accuracy risk, and there's no calibration set to build and maintain. Reach for int8 when the hardware forces it or when fp16 doesn't get you inside the budget.

Where this stops being right

• A task needing fine detail — reading small text, not spotting a gap — can lose too much at int8; use fp16 or full precision there.
• Hardware without int8 acceleration — quantization buys a smaller model but not much speed.
• A low-stakes deployment — over-investing in quantization-aware training isn't worth it; fp16 plus a wide human-review band works.

FAQ

Does quantization make the model less accurate? fp16 is near-lossless for most detection models. int8 has a small cost you measure against your evaluation set — usually acceptable for gap detection.

What's a calibration set? A representative sample of real images the model runs through to set the int8 scaling. Use real store photos, not synthetic.

When do we need quantization-aware training? When the post-training int8 accuracy hit is too large. It simulates quantization during training for a better result, at more effort.

How much smaller does the model get? int8 is roughly a quarter the size of a 32-bit float model; fp16 is half. On a memory-constrained accelerator that reduction can be the difference between the model fitting on the device and not.

ISTRALLEN quantizes edge detection models to fit the accelerator and measures the accuracy cost against real store photos; see AI for Retail.