Barge-In: How a Voice Agent Handles Being Interrupted Mid-Sentence

The interruption that has to just work

A caller starts talking while the agent is still speaking — cutting off a long confirmation, correcting a detail, or saying "no, wait." Voice AI barge-in is the agent stopping, listening, and responding to the new input the way a person would. An agent that talks over the caller, or ignores them and finishes its scripted line, reads as deaf within the first exchange.

Why it's not optional

Interrupting is ordinary phone behaviour. People cut in constantly, and they expect the other party to yield. On a live call there's no way to "scroll back," so an agent that can't be interrupted forces the caller to sit through output they've already decided they don't want.

The pieces that make it work

• Voice activity detection — is the caller actually speaking, or is that a cough, a door, or background TV?
• Endpointing — when has the caller finished their turn, so the agent knows it's time to respond rather than wait for more?
• Cancelling in-flight output — stopping the audio that's already streaming to the caller, and, in a native speech-to-speech session, halting generation partway.

Why cascaded pipelines struggle

A cascaded speech-to-text then LLM then text-to-speech stack is already streaming a synthesized response to the caller. To handle an interruption it needs a separate layer to detect incoming speech and kill the playback, and then the language model has to re-plan around a half-delivered answer. On our voice AI engagement the native speech-to-speech session processes audio in and out together and reacts to incoming speech in the same session, without a separate coordination layer to keep in sync.

The tuning trade-off

Too eager, and every "mm-hmm" or bit of background noise cuts the agent off mid-word. Too conservative, and real interruptions get ignored. It comes down to voice-activity-detection sensitivity and a short grace period before the agent treats incoming audio as a genuine turn.

There's also a content dimension: a backchannel like "uh-huh" or "right" while the agent is talking usually means "keep going," not "stop." A system that treats every sound as a full interruption feels twitchy. Distinguishing a backchannel from a real turn — by length, by prosody, or by whether the caller keeps talking — is part of getting barge-in to feel natural rather than jumpy.

Echo, the other failure mode

If the agent hears its own output through the telephony leg, it can trigger a false barge-in and talk itself into a stutter. Echo cancellation on the phone leg is part of making barge-in reliable, not a separate concern.

How to test it

Barge-in doesn't show up in a happy-path demo — you have to provoke it. Record test calls with deliberate interruptions: a caller correcting a detail mid-sentence, a caller cutting off a long confirmation with "yes, that's right," a cough, a burst of background TV, someone else in the room talking. For each, check two things: did the agent yield when it should have, and did it false-trigger when it shouldn't. Track the yield rate and the false-barge rate as first-class metrics, and watch whether calls with interruptions have worse satisfaction than calls without — if they do, the tuning isn't there yet.

Where this stops being right

• Legally required statements. A scripted disclosure or a regulated notice may need to play to completion — some systems suppress barge-in for those specific segments only.
• Very noisy caller environments make voice activity detection unreliable; a push-to-talk or keypad fallback can help.
• A status-only bot with short one-line answers barely benefits from barge-in — there's little to interrupt.

FAQ

Is barge-in just voice activity detection? Voice activity detection is one piece. You also need endpointing and the ability to cancel output that's already playing.

Why can't our cascaded bot do it well? It has to detect speech and kill a streaming text-to-speech response mid-playback, then re-plan. A native speech-to-speech session reacts within the same session.

Should the agent ever ignore an interruption? Only during a legally required disclosure that must be delivered in full. Otherwise, no — yielding is the expected behaviour.

ISTRALLEN builds voice agents on native speech-to-speech so a caller can talk over the agent the way they would a person; see AI for Fintech.