This is the depth of analysis behind every project we build. We publish it as confirmation of how we actually work — the real options we weighed, the numbers that ruled them out, the architecture that survived contact with a live phone line — so our process stays as transparent to clients as the results are. Voice is a different kind of "carving from stone" than a chat interface: a customer can’t see a typing indicator or scroll back to reread something, so every latency and hand-off decision below had to survive contact with an actual phone call, not a demo.
The client was a consumer lender whose call center handled first-line loan application calls — "what’s my rate," "where’s my application," "can I qualify for X." Call volume had grown faster than headcount, and the same small set of questions was eating most of an agent’s day before they ever got to the calls that actually needed a person: judgment calls on borderline applications, negotiating terms, handling disputes.
The ask was not "replace the call center with a bot." It was: handle the high-volume, low-complexity first-line calls end to end over the phone, and hand off anything touching an actual credit decision to a human — without the caller having to repeat themselves. That handoff requirement mattered as much as the automation itself; a voice agent that saves time on 80% of calls but makes the other 20% worse by making the customer start over is a net loss, not a win.
The second constraint, non-negotiable from day one: this is a regulated lending workflow. A credit decision has to be explainable to the applicant and reconstructable for a regulator, and that requirement shaped the escalation architecture more than any latency number did.
Why the cascaded pipeline’s latency stacking was the core problem: OpenAI’s own May 2024 announcement of GPT-4o gives a directly relevant before/after comparison from their own product — their prior cascaded voice mode in ChatGPT, three separate models chained for transcription, text generation, and speech synthesis, averaged 2.8 seconds of latency on GPT-3.5 and 5.4 seconds on GPT-4, against GPT-4o’s native audio responding in as little as 232ms and averaging 320ms, which OpenAI describes as comparable to human response time in conversation. That’s OpenAI comparing their own old and new products, not an independent benchmark, but the architectural reason for the gap is the relevant fact: a cascaded pipeline pays the latency of three sequential network calls, where a native speech-to-speech session pays the latency of one. Independent engineering write-ups from Twilio’s and Telnyx’s own blogs on voice AI latency put a well-optimized cascaded stack in the 600ms-1.7s range end-to-end — still measurably slower than native speech-to-speech, and with more moving parts to keep fast.
Why interruption handling, not just raw latency, was the deciding factor: a caller correcting themselves mid-sentence is completely ordinary phone behavior. A cascaded pipeline generally has to finish playing its current TTS output (or bolt on a separate interruption-detection layer) before it can process new input; OpenAI’s Realtime API, launched in public beta October 1, 2024, handles barge-in natively as part of the same speech-to-speech session, the same interruption-handling behavior as ChatGPT’s Advanced Voice Mode. We’re noting a limit to our own confidence here: we weren’t able to confirm an independently published latency figure specifically for the Realtime API endpoint itself — the honest statement is that it inherits the same speech-to-speech architecture, not that we’re quoting an official Realtime-API-specific number.
Why not self-hosted open-source speech-to-speech: this is the same build-vs-buy tradeoff we’d apply to any latency-critical, regulated workload on a tight timeline — owning GPU inference, model updates, and the interruption-handling behavior ourselves is a real infrastructure commitment, and on an 8-week build window with no dedicated speech-ML function on this project, that commitment doesn’t pay for itself against a managed API with the interruption handling already solved. We’d revisit this if call volume grew large enough that per-minute API cost overtook the infrastructure and ops cost of self-hosting.
OpenAI’s Realtime API, a single native speech-to-speech session rather than a chained pipeline — the architecture choice that everything else in this document assumes.
Why Media Streams’ bidirectional support was the specific technical fit: Twilio’s own documentation distinguishes `<Start><Stream>`, which only streams call audio out to your application, from `<Connect><Stream>`, which is bidirectional — audio flows both directions over the same WebSocket. That bidirectional path is what lets raw call audio flow directly into and out of a Realtime API session without an extra translation or buffering layer in between, which matters directly because of the architecture chosen in §1: a native speech-to-speech session is only as fast as the audio pipe feeding it.
Why Twilio’s operational maturity mattered on a regulated, customer-facing phone line: Twilio’s own published SLA commits to 99.95% uptime standard, with 99.99% available on Enterprise/Admin Edition. Independent analyst research from Metrigy’s CPaaS Market Share & Forecast (3Q24) put Twilio’s CPaaS revenue share at roughly 23.2% in 2023, growing to roughly 25.9% by Q3 2024 — the largest player in the category by that measure, which we’re citing as third-party market research, not Twilio’s own marketing. For a phone line customers already call to reach a lender, betting on the platform with the deepest operational track record was a reasonable default over betting a regulated customer channel on a smaller, less-proven integration path.
Pricing, and an honest gap in what we could verify: Twilio’s official US pricing page lists outbound voice around $0.013/minute for the base voice product. AI-specific per-minute pricing figures (commonly cited around $0.07/minute across secondary aggregators) weren’t something we could confirm against Twilio’s own pricing page directly, and pricing varies by direction and country regardless — we’re flagging this as a number to re-verify at build time, not one to quote precisely to a client today. We also didn’t find a direct, apples-to-apples latency or reliability comparison against Amazon Connect or Vonage for this specific use case, and don’t want to fabricate one — the honest version of this decision is that Media Streams’ bidirectional WebSocket support was the most direct integration path to the architecture already chosen in §1, combined with Twilio’s market-leading maturity and a contractual SLA, and that combination was enough to not need that comparison to be confident in the call.
Twilio, using Media Streams’ bidirectional audio, bridging the PSTN call directly to a Realtime API session over WebSocket.
Why the standard text-chat tool-calling latency budget doesn’t transfer to voice: a text chat interface can tolerate a multi-second pause for a tool call because there’s a visible "thinking" or typing indicator and no live-audio expectation to break. A phone call has no equivalent affordance — silence on a live call reads as "did this drop" almost immediately. Stivers et al.’s 2009 PNAS study on turn-taking in conversation, which measured response gaps across ten languages, found an average cross-linguistic response offset of roughly 208 milliseconds, with individual language averages clustering within about 250ms of that figure. We’re citing this deliberately as human-to-human conversational research, not a study of AI voice agents — treating "keep function calls fast enough to not obviously break natural turn-taking" as our own engineering inference from that research, not a conclusion the paper itself draws.
Why pre-fetching everything up front doesn’t actually solve the problem: the information a caller needs isn’t known until they identify themselves partway into the call, so there’s nothing meaningful to pre-fetch before the conversation starts — the lookup has to happen live, inside the call, which is exactly the case the latency budget above has to cover.
Why session-level function calling, specifically, was the fit: OpenAI’s Realtime API documentation describes tools configurable at the session level or per-response, with the model able to signal a function call in the middle of generating a response and incorporate the result into the continuing audio reply, rather than the call blocking the entire turn. The practical consequence for our side of the integration: the Postgres queries backing these functions had to be simple, indexed point lookups — not open-ended reporting queries. The latency budget here is set by conversational norms, not by what the database can technically handle at the low end.
Session-level function calling against narrowly-scoped, indexed Postgres lookups, with the tool-call latency budget treated as a conversational-quality requirement, not just a technical performance target.
Why letting the agent make the actual credit decision wasn’t on the table, as a matter of regulation rather than caution: the Equal Credit Opportunity Act and its implementing Regulation B (12 CFR § 1002.9) require a creditor to issue an adverse action notice within 30 days of a credit denial, citing the specific principal reasons that genuinely reflect the factors the decision was based on. The CFPB’s own Circular 2022-03, published September 2022, states explicitly that this obligation applies to decisions made using complex algorithms, and that a creditor cannot use the complexity of the algorithm itself as a defense for failing to provide specific, accurate reasons. That’s a regulatory constraint on the architecture, not a risk-tolerance judgment call — anything that could constitute an adverse action needs a traceable, human-explainable reasoning path, which rules out the model being the final decision-maker.
Why async handoff undercuts the point of the project: an async handoff means the caller hangs up without knowing their status, and a callback later loses the context and momentum that made the call efficient to begin with — the applicant often ends up re-explaining details they already gave the agent. That doesn’t reduce handling time, it defers it and adds a second call, which works directly against the reason the project exists.
Why live warm handoff, specifically, was the fit — and what it required elsewhere in the architecture: recognizing mid-call that a case needs a human, then transferring the live call with the transcript and structured application data already loaded on the receiving agent’s screen, means the customer never repeats information already given to the AI. The requirement this drives directly: the data layer (§5) has to make the full call context available to a human agent’s console inside roughly the same window it takes the phone system to complete the transfer — not a batch export that catches up a few minutes later.
Live warm handoff, triggered automatically once the conversation crosses into territory that could constitute an adverse action, with the human loan officer’s console pre-loaded with the transcript and structured data by the time the call connects.
Why in-memory-only session state fails exactly at the moment it matters most: it doesn’t survive the one event the whole architecture is built around — the live warm handoff in §4. If session state lives only inside the voice agent’s own process, and the human agent’s console reads from a separate system, "the transcript is already loaded when the call connects" becomes a race against however long it takes to flush, batch, or sync data between two systems — reintroducing exactly the staleness risk this component exists to eliminate.
Why a separate, dedicated analytics store wasn’t the right shape either: these transcripts aren’t just a record to review later — they’re operational data a human loan officer has to act on inside the same call, and they need to line up exactly with the application data already sitting in the client’s loan origination system. Running that as a second, parallel store that can silently drift from the system of record is a bigger risk in a regulated lending context than the operational simplicity of a dedicated logging pipeline is worth.
The compliance requirement that shaped this, specifically: the Telephone Consumer Protection Act requires disclosed consent before recording a call, and in twelve U.S. states with all-party or two-party consent rules specifically — a detail we’re flagging as needing case-by-case legal confirmation for the client’s actual calling footprint, not a blanket rule we’re treating as settled everywhere. That consent record, together with the full transcript, needs to be durably stored and queryable for as long as the applicable retention window requires.
Postgres, shared with the existing loan origination system, as the system of record for live session state used during warm handoffs and for durable transcripts and consent records used for compliance retention — not a parallel, disconnected data store.
Both published numbers are downstream of the decisions above, but bounded by one of them in particular:
We’re not aware of a published third-party figure for "average handle time for an automated loan-status call" to compare against, and we’re not implying one exists — these two numbers stand on their own as project-specific results.
Worth stating plainly, because no architecture is permanent:
None of these are failures of the original decision — they’re the conditions under which the same reasoning process, run again, would produce a different answer.
We’re publishing this confidence table on purpose. A client is better served by knowing which numbers came straight from a primary source and which ones we’d re-verify — especially the legal and compliance claims here, which deserve a lawyer’s sign-off for a specific client’s footprint, not just our own reading of secondary summaries — than by a document that reads clean because the uncertainty got quietly edited out.