Streaming Tokens While Running a Tool Call: Keeping Support Chat Live

Perceived latency is the whole game

A support chat feels broken if the first token takes three seconds, or if the window sits frozen while a lookup runs. A streaming AI support agent solves both: it sends tokens as the model generates them, and it narrates the pause when it has to stop and check something.

Token streaming

Instead of waiting for the full response and sending it as a block, the agent streams each token as it's produced. The customer sees a reply forming within a few hundred milliseconds, even though the complete answer takes a couple of seconds. The wait is the same; the experience is not.

The mid-stream tool call

The interesting part is what happens when the agent needs data partway through a sentence. On our support agent project the loop holds the chat connection open, streams tokens as the model generates them, and — mid-stream — pauses to execute a tool call against the real order or returns API before resuming the response with the result folded in. The customer sees the reply start, pause briefly, and continue with their tracking date rather than a generic one.

Narrate the wait

A short acknowledgement before a slow tool call — "let me pull up that order" — turns a silent pause into an expected one. Silence during a lookup reads as a freeze; a sentence of narration reads as someone checking.

Why this needs an async backend

The chat connection is held open and mostly idle — waiting on the model, on the database, on the client's order webhook. An async event loop holds hundreds of these on one process, because a waiting request yields instead of holding a thread. A synchronous backend holds one per worker for the duration. Streaming a support agent at any real concurrency assumes async serving underneath.

Parallelise the reads

When the agent needs the order, the tracking record, and the policy, fetch them at once, not one after another. The pause is then a single round-trip instead of three, and the customer barely notices it.

The transport

Streaming tokens to the widget is usually server-sent events or a WebSocket. Server-sent events are simpler for one-directional token streaming and reconnect cleanly; a WebSocket is worth it if you need the client to send events back mid-response. Either way the widget has to render a response that's still growing and handle a mid-stream pause without looking stuck — a small blinking indicator during a tool call does the job.

What the customer actually sees

A concrete timeline for "where's my order": the reply starts forming in roughly 300 milliseconds. It writes a sentence, then pauses with "let me pull up that order" for a second or two while the tool call runs. It resumes with the real tracking status and delivery date. Total elapsed time is a few seconds — the same as waiting for a block response — but the window is never silent, and that's the difference between "this is working" and "did it break."

Where this stops being right

• If your latency is already low — fast model, fast tools — a plain request/response is simpler and the streaming complexity isn't worth it.
• Very fast tool calls don't need the narration; save it for the ones that might be slow.
• Async channels like email have no latency pressure — none of this applies there, and a batch reply is fine.

FAQ

Why stream instead of sending the whole response at once? A chat that shows nothing for three seconds reads as broken. Streaming makes the first token near-instant.

What happens to the stream during a tool call? It pauses, the tool runs, and it resumes incorporating the result — ideally with a short "checking..." so the pause is expected.

Does streaming need a special backend? An async one. The connection is held open and mostly idle; an async event loop serves many at once where a sync worker serves one. Trying to stream at real concurrency on a synchronous framework means a worker process per open chat, which doesn't scale past a small number.

ISTRALLEN builds support agents that stream tokens and run tool calls mid-response so the chat never goes silent; see AI for E-commerce.