Picking a Confidence Threshold: When Should AI Escalate to a Human?

The decision that keeps automation from backfiring

An AI system that acts on everything is easy to build and usually a bad idea. The hard part is knowing when not to act — routing the shaky and the sensitive cases to a person. Picking an AI confidence threshold is how you draw that line, and the common mistakes are using one cutoff where you need two, and trusting a score that isn't calibrated.

"Confidence" isn't one number

A raw model probability is often overconfident, and on its own it's a weak signal. Real routing combines several:

• Model uncertainty — self-reported, or behavioural (hedging, contradicting itself, repeated clarification).
• Tool and retrieval health — a lookup errored, timed out, or came back empty; the retrieved passages are weak matches.
• Input quality — a photo with glare, a garbled transcript, a malformed field.
• Stakes and topic — a refund over a threshold, an adverse-action-adjacent decision, anything legal or safety-related escalates regardless of score.

Two cutoffs, not one

The pattern that fraud scoring, support agents, and shelf monitoring in our portfolio all converged on is three zones, not a single threshold: a confident-handle zone, a confident-act (or confident-block) zone, and an ambiguous band in the middle that goes to a human. One cutoff forces every borderline case to resolve as a coin flip dressed up as a decision.

Calibrate before you trust a number

If the score isn't calibrated, "0.8" doesn't mean 80%. Bin your scores against actual outcomes, or apply temperature scaling, so that a threshold you set actually corresponds to the error rate you think it does. Setting a threshold on an uncalibrated score is guessing with extra steps.

The quick check: pull a few hundred decisions, group them by score band, and compare the score to the realised outcome rate in each band. If the 0.7-0.8 band actually resolves correctly 55% of the time, your scores are optimistic and every threshold you've set is looser than you think. Re-check this after any model change — calibration drifts.

Set it conservative, then tune

Start by escalating more than you think you need to — you can't un-ring a bad automated decision. Then measure two things against each other: the automation rate, and the quality of the automated cases (CSAT, false-positive rate, alert hit rate). Loosen the threshold only while automated-case quality holds. If it drops, you've automated cases you shouldn't have.

Turn the signals into a decision

A workable structure is a policy layer sitting on top of the model:

1. Hard rules first. Money above a limit, legal or safety topics, an explicit request for a human — straight to a person, no scoring. These aren't confidence calls.
2. A weighted combination for the soft signals — model uncertainty, retrieval health, sentiment, input quality — producing the escalate/continue decision for everything else.
3. Per-topic thresholds on top, so "where's my order" and "process my refund" don't share a bar.

Per-stakes thresholds

A "where's my order" question can tolerate a lower bar than "process my refund." A clear shelf photo can auto-alert where a promo-packaging shot goes to review. Vary the threshold by what a wrong call costs.

Escalations are training data

Every escalation is a labelled example of something the system couldn't handle confidently. Tag them by reason — weak retrieval, an unhandled intent, a tool that returned empty, low model certainty — and review the buckets weekly. A spike in one bucket tells you what to fix next, and whether the threshold is doing its job or just papering over a gap.

Where this stops being right

• No human capacity to escalate to. The threshold degrades to "the system gives up" — fix the staffing first.
• Every case is low-stakes. A single cutoff is fine; the two-zone structure is overhead.
• Tiny volume. A scoring policy is over-engineering — "escalate anything transactional" is enough.

FAQ

Can I just use the model's confidence score? As one input, and only after calibrating it. Raw scores are usually overconfident, especially from LLMs.

One threshold or two? Two — a confident-yes zone, a confident-no zone, and a human band between them.

How do I know my threshold is too loose? Automated-case quality drops while the automation rate climbs. That's the signal you've pushed the band too wide.

ISTRALLEN builds AI that routes the confident cases automatically and sends the ambiguous band to a person — see what we do.