AI Incident Response: What to Do When the Model Is Wrong at Scale

An AI incident isn't a normal outage

The service is up. Requests return 200. But the answers are subtly wrong, or a prompt change is quietly declining valid cases, or a tool started returning empty and the model is improvising around it. AI incident response is the runbook for "the model is wrong at scale," and it's different from a crash because nothing is obviously broken.

Detect

The signal comes from observability, not a health check: a spike in the refusal rate, the tool-error rate, the unsupported-claim rate, the escalation rate by reason, or cost per task. If those aren't instrumented with alert thresholds, you find out from customer complaints — which is late.

Triage: where is it wrong?

Walk the trace of a bad interaction and isolate the layer:

• Upstream data. A source system changed a field, went stale, or started returning nulls. The model is doing its best with bad input.
• Retrieval. The index is stale or a re-index job failed; the right passage isn't being retrieved.
• A tool. An API is erroring or timing out and the failure is being swallowed.
• The prompt. A recent edit changed behaviour in a way the eval set didn't catch.
• The model. A provider-side model update shifted behaviour, or your fine-tuned model drifted.

Most incidents are one of the first three — upstream of the model.

Contain

Don't debug in production with real users taking the hit. Contain first:

• Feature-flag the affected path back to rules, to a human queue, or off.
• Roll back the prompt version or the model snapshot to the last known-good.
• Widen the escalation band so more of the ambiguous traffic goes to people while you fix it.

On the projects in our portfolio every automated path has a flag that routes it back to a human or a deterministic fallback — containment is a config change, not a deploy.

Communicate and root-cause

Tell the affected teams what's degraded and what the fallback is. Then find the cause from the traces, fix it, and — this is the part that's usually skipped — add the failure as a new case in the evaluation set so the same thing can't ship again silently.

A worked incident

The refusal rate on a support agent doubles overnight and the alert fires. You pull the trace of a refused-but-valid interaction: retrieval returned an empty set. You check the re-index job — it failed two nights ago, so the index is stale, and for anything the catalogue added since, there's simply no passage to retrieve, and the relevance floor is correctly declining. Contain: feature-flag the affected intents to the human queue. Fix: re-run the index, add an alert on re-index-job failure, and add "retrieval returned empty" as a case in the eval set. Total customer-facing impact: a few hours, because an alert caught it — not a week, because a customer did.

The postmortem question

The one question worth asking every time: did an alert catch this, or did a customer? If a customer, the fix isn't only the bug — it's the missing metric or threshold that should have paged someone first.

Where this stops being right

• A low-stakes internal tool — the runbook can be "turn it off and look at the logs."
• A feature with no automated action — containment is simpler; there's less to roll back.
• Extremely low volume — you'll catch issues by eye before an alert would fire; formal incident response is overhead.

FAQ

How do we even know an AI incident is happening? From observability metrics with alert thresholds — refusal rate, tool-error rate, unsupported-claim rate, cost per task. Without those, you learn from complaints.

What's the first move? Contain, not debug. Feature-flag the path back to rules or humans, or roll back the prompt or model, before you investigate.

Where are AI incidents usually caused? Upstream of the model — a changed source field, a stale index, a silently failing tool — more often than the model itself. Which is why the trace has to show retrieval results and raw tool responses, not just the model's input and output.

ISTRALLEN builds AI features with a containment flag on every automated path and traces that make root-cause fast — see what we do.