LLM Observability: What to Log and Alert On

You can't debug what you didn't record

An LLM feature fails in ways a normal service doesn't — a subtly wrong answer, a refused-but-valid request, a tool call that shouldn't have fired. LLM observability is recording enough of each interaction to answer "what happened here" after the fact, and alerting on the patterns that mean something's wrong.

Trace the whole request

For each interaction, capture the full chain:

• The assembled prompt — system prompt version, the retrieved passages, the tool results, the user text, clearly separated.
• The tool calls — which, with what arguments, what raw result, how long.
• The raw model output — before any parsing or post-processing.
• Tokens and cost per call, and per interaction.
• Latency per step, so a slow tool or a slow model call is visible.

This is verbose. Keep full traces for a sample plus every escalated or badly-scored interaction, and lighter records for the rest.

The metrics to alert on

• Refusal / "I don't know" rate. A spike means retrieval degraded or the prompt changed.
• Tool-call error rate. A downstream API is failing, or a schema drifted.
• Escalation rate, by reason. A jump in "low confidence" points at thin coverage; a jump in "tool returned empty" points at a data problem.
• Unsupported-statement rate. Claims in the output not backed by a retrieved source or a tool result — should sit near zero by design.
• Cost per task and calls per task. A quiet drift up is the invoice warning you before the invoice does.
• p95 latency, end to end and per step.

Link it to outcomes

Thread a conversation ID through the transcript, the tool records, the escalation payload, the satisfaction survey, and any re-contact. Then "did this actually resolve" is answerable, and you can compare traces for good and bad outcomes.

The connection to shadow mode and drift

The same trace store powers pre-launch shadow testing and ongoing drift monitoring — you're comparing a new component's outputs, or this month's outputs, against a recorded baseline. On our fraud-scoring engagement analyst decisions are written back to the event log as labelled outcomes, so the trace of every borderline case has a verdict attached.

Reading a bad interaction

When a complaint comes in, the trace is what turns "the bot was wrong" into a fix. You pull the interaction by conversation ID and walk it: what did retrieval return — was the right passage even in the set? What did the tools return — did one error silently? What was the raw model output before parsing — did the post-processing mangle a good answer? Which prompt version produced it? Nine times out of ten the failure is upstream of the model, and without the trace you'd be guessing.

Where the traces power the rest

The same store feeds three other things: pre-launch shadow testing (compare a new component's traced outputs against the recorded baseline), drift monitoring (compare this month's distribution against last month's), and the evaluation set (mine real traces for the hard cases worth adding). On our fraud-scoring engagement analyst verdicts on borderline cases are written back to the event log, so the trace of every ambiguous decision carries a labelled outcome — which is both an audit record and training data. The projects in our portfolio treat the trace store as core infrastructure, not an add-on.

Where this stops being right

• A low-stakes internal tool — conversation logs plus basic latency and error metrics may be enough.
• Extreme volume — full-trace sampling has to be tuned so the observability store doesn't cost more than the feature.
• Strict data-minimisation requirements — traces contain user input; apply the same retention and access controls as any other store of that data.

FAQ

What's the one thing to always capture? Every tool call with its arguments, result, and the model's stated reason — that's the record you pull when someone asks "why did it do that."

What should page someone at 3am? A spike in tool-call errors or the refusal rate, and cost per task drifting well above budget.

How much of this do we keep? Full traces for a sample and for every escalated interaction; lighter records for the rest, under normal retention rules.

ISTRALLEN builds LLM features with end-to-end tracing and alerting on refusals, tool errors, cost, and drift — see what we do.