How a Low-Latency Voice AI Assistant Is Architected

The requirement that becomes an architecture problem

Human conversation has a turn-taking rhythm measured in a couple hundred milliseconds. A voice agent that pauses for seconds doesn't read as thoughtful, it reads as broken. That single constraint drives most of a voice AI assistant architecture — the model choice, the telephony bridge, how tool calls are timed, and where session state lives.

Cascaded pipeline vs native speech-to-speech

The traditional way to build a voice bot chains three services: speech-to-text, then an LLM, then text-to-speech. Each is a network round-trip, and the latency stacks. OpenAI's own comparison of their prior cascaded voice mode against GPT-4o's native audio put the gap at seconds versus a few hundred milliseconds — the architectural reason being one round-trip instead of three. Well-optimised cascaded stacks still land in the several-hundred-millisecond to multi-second range end to end, with more moving parts to keep fast.

Our voice AI engagement uses a single native speech-to-speech session for exactly this reason — it's the decision everything else in the design assumes.

Barge-in

A caller correcting themselves mid-sentence is ordinary phone behaviour. A cascaded pipeline generally has to finish playing its current output, or bolt on a separate interruption-detection layer, before it can react. A native speech-to-speech session handles the interruption as part of the same session. If your agent can't be talked over, callers notice within the first exchange.

The telephony bridge

The call arrives over the phone network; the model session speaks a different protocol. You need bidirectional audio streaming — typically a WebSocket carrying audio both directions — bridging the live call into and out of the model session. The audio pipe has to be as fast as the model, or it becomes the bottleneck the model choice was meant to avoid.

Function calling under a conversational budget

The agent needs to look things up mid-conversation — application status, account details — without dead air. That means session-level tool calls the model can trigger mid-turn and keep talking while the result comes back. The practical constraint on your side: the queries behind those tools have to be narrow, indexed point lookups, not open-ended reporting queries. The latency budget here is set by conversational norms, not by what the database can technically handle.

Pre-fetching everything up front doesn't rescue you, because the thing the caller needs isn't known until they identify themselves partway into the call. There's nothing meaningful to load before the conversation starts. The lookup has to happen live, inside the call, which is exactly the case the latency budget has to cover — so the fix is fast queries and, where a specific integration is genuinely slow, a cache in front of it, not a redesign that pretends the lookup can be avoided.

State that survives the handoff

Anything touching an actual credit decision hands off to a human. For that handoff to be "warm" — the caller doesn't repeat themselves — the transcript and collected data have to be on the human's screen by the time the call connects. If session state lives only inside the voice agent's own process, and the human console reads from a separate system, you've reintroduced a sync race exactly where it hurts most. The session and the durable transcript belong in a store both sides read.

Where this stops being right

• You may not need real-time. If a callback-based flow is acceptable, a cascaded pipeline is simpler and cheaper.
• A single slow integration doesn't justify re-architecting — that specific tool call needs pre-fetching or caching.
• Multi-language needs its own review of the speech model's language and voice coverage; don't assume it.
• Very low call volume doesn't amortise the build — the fixed cost dominates.

FAQ

Cascaded or native speech-to-speech? Native for a live phone line where turn-taking latency is the constraint. Cascaded is fine when the latency tolerance is higher or you need to mix specific components.

Where does the function-call latency budget come from? Conversational turn-taking — roughly a couple hundred milliseconds — not from what the database can do at the low end.

Can it be interrupted? With a native speech-to-speech session, yes, as part of the session. A cascaded pipeline needs extra work to handle barge-in.

ISTRALLEN builds low-latency voice agents for fintech call centres — native speech-to-speech, warm handoff, and grounded lookups as the core of the design; see AI for Fintech.