Human-in-the-Loop AI: Where to Put the Person
"Add a human" isn't a design
Everyone agrees an AI system needs a human somewhere. Where, exactly, is the decision that matters. Human-in-the-loop AI has four common placements, and they trade latency, cost, and risk against each other differently.
Before the action: approval
The AI proposes, a person approves or rejects before anything happens. Highest safety, highest latency and cost — you're adding a human step to every affected case. Right for high-stakes, low-volume actions: a large refund, a credit decision, an enforcement action against a seller. On our support-agent engagement, write actions above a value threshold queue for a human to approve with one click, rather than the agent executing on its own judgement.
After the action: review
The AI acts, a person reviews afterward and can reverse. Low latency, but the mistake already happened. Fine when the action is cheap to reverse — a tag, a routing decision, a draft. Wrong when the action is money moving or a customer being declined, because by review time the funds have settled.
On the exceptions: a routed queue
The AI handles the confident cases automatically and routes only the ambiguous ones to a person. This is the pattern fraud scoring, support agents, and shelf monitoring in our portfolio all converged on — the confident majority automated, the uncertain minority to a human review queue. It needs calibrated confidence and a staffed queue, and it's the best balance for high-volume decisions with a meaningful grey zone.
On a sample: spot-check
A person audits a random fraction of automated outputs to catch drift and systematic errors. Cheapest, catches nothing in real time. A complement to the other three, not a replacement — it's how you know the automated path is still trustworthy.
The handoff is part of the design
Wherever the human sits, they need context — the transcript, the data, the model's reasoning — not a raw score and a blank screen. A human queue where every case is a fresh investigation is a bottleneck, not a safeguard.
A quick decision table
| Action | Volume | Reversible? | Placement |
|---|---|---|---|
| Large refund, credit denial, seller ban | Low | No | Pre-action approval |
| Ticket tag, routing, a draft reply | High | Yes | Post-hoc review |
| Fraud decision, support resolution, shelf alert | High | Partly | Exception queue |
| Everything automated | — | — | Spot-check on top |
Most real systems run three of these at once: an exception queue for the grey zone, pre-action approval for the biggest money moves, and a spot-check over the whole automated path.
The cost of the human sitting in the wrong place
Put the person before every action and you've capped throughput at human speed and paid for it. Put them only on a post-hoc sample and a systematic error runs for a week before anyone notices. The placement is a real trade-off, and it's worth deciding deliberately rather than defaulting to "we'll review everything."
As the model improves, the placement can shift. A feature that launches with pre-action approval on most cases can move to an exception queue once the confidence scores are calibrated and the error rate is proven — the human's role narrows from "check everything" to "handle the genuinely hard cases," and the throughput ceiling lifts with it.
Where this stops being right
- No human capacity — every placement assumes someone to escalate to. Without staffing, it degrades to "the AI gives up."
- Genuinely low-stakes, high-volume output — a spot-check may be all it's worth.
- A hard latency budget — pre-action approval doesn't fit a real-time path; use the exception queue with a tight confidence band.
FAQ
Which placement is the default? The routed exception queue for high-volume decisions with a grey zone. Pre-action approval for the high-stakes, low-volume actions on top of it.
Can we use more than one? Usually you should — an exception queue for the ambiguous cases, pre-action approval for large money actions, and a spot-check over everything to catch drift.
What does the human need to decide fast? The full context and the model's reasoning on one screen, with one-key actions. Not a score and a form to fill in. The single biggest factor in what a review queue costs is how many seconds it takes to clear one item, and that's decided by the interface, not the model.
ISTRALLEN designs where the human sits — approval, review, exception queue, or spot-check — against the stakes and the volume; see what we do.