Confidence Calibration for a Detection Model: Making the Score Mean Something

The routing depends on the score meaning what it says

Confidence-based routing sends high-confidence detections straight to an alert and the ambiguous middle band to human review. Detection model confidence calibration is what makes that split trustworthy — because a raw model score usually doesn't mean what it looks like it means.

The problem

A detection model's raw confidence output is not a probability. A "0.9" doesn't mean "right 90% of the time." Models are often overconfident, and the miscalibration varies by class, by lighting, and by how far the input is from the training distribution. On messy real store photos it's usually worse than on a clean benchmark.

Why it matters here specifically

You're setting a threshold that decides auto-alert versus human review versus ignore. If 0.9 actually corresponds to 70% correct, your auto-alert band is full of wrong alerts, and store teams learn to tune the feed out. The threshold is only as good as the calibration of the number under it.

How to check calibration

A reliability diagram: bin detections by confidence, and for each bin compare the average stated confidence to the actual accuracy in that bin. A well-calibrated model's line sits on the diagonal; an overconfident one sags below it — its 0.9 bin is only 0.7 accurate.

How to fix it

Temperature scaling — a single learned parameter that rescales the model's outputs, fit on a held-out labelled set. Simple, cheap, and usually enough.

Per-class and per-condition thresholds

Calibration isn't uniform. A model can be well-calibrated on clear front-facing products and badly calibrated on occluded ones. Where you have the data, set the routing thresholds per class or per capture condition rather than one global number.

Validate on real store photos

Calibrate and check on the messy real inputs — glare, angles, promo packaging — not a clean set. On our computer vision project the evaluation data was real photos from the hardest stores, and calibration belongs on that same distribution or the thresholds are tuned for conditions you don't have. Re-check after every retraining — calibration drifts when the model changes.

A concrete calibration workflow

Hold out around 1,000 labelled detections from varied stores. Sort them into 10 confidence buckets — 0.0-0.1, 0.1-0.2, and so on. For each bucket, plot the average stated confidence against the actual accuracy. If the 0.9 bucket is only 72% accurate, the model is overconfident. Fit one temperature parameter on that held-out set to pull the curve toward the diagonal, re-plot to confirm, and then set your auto-alert threshold on the calibrated score — so "calibrated 0.9" genuinely means about 88% correct, and the store team's trust in the feed is earned.

Why per-condition calibration matters

Split the reliability diagram by capture condition — clear front-facing detections versus occluded or glare-affected ones. If the model is well-calibrated on the first and badly on the second, a single global threshold is wrong for half your inputs. Route the occluded detections to human review at a lower bar, or calibrate them separately.

Calibration is not accuracy

Calibrating a model doesn't make it more accurate — it makes its confidence number honest about how accurate it is. A well-calibrated model can still miss detections; it just won't claim 0.95 confidence while doing it. You need both: a model accurate enough for the job, and a confidence score you can set routing thresholds on. Calibration fixes the second, not the first, and it's easy to conflate them when a "calibrated" model still isn't good enough.

Where this stops being right

• Very class-dependent miscalibration — temperature scaling assumes it's roughly uniform; you may need per-class scaling instead.
• No labelled held-out set — you can't calibrate against nothing; it's the same cost as the evaluation set.
• A low-stakes deployment — a rough threshold plus a wide human-review band works without formal calibration.

FAQ

Is the model's confidence score a probability? No — it's a raw output that's usually overconfident. Calibrate it before you set routing thresholds on it.

How do we check calibration? A reliability diagram — bin by confidence and compare stated confidence to actual accuracy per bin.

How do we fix it? Temperature scaling — one learned parameter fit on a held-out set — is simple and usually enough. Add per-class thresholds where calibration varies by condition.

ISTRALLEN calibrates detection models on real store photos so confidence routing thresholds mean what they say; see AI for Retail.