Measuring Speech Recognition Accuracy by Accent and Language

The blended number hides the problem

A speech model can be 95% accurate overall and 70% for one accent group, and the aggregate will look fine because it's weighted by your current caller mix. Voice AI accent accuracy has to be measured per segment, or the callers the agent serves worst are exactly the ones you never see in the metric.

Word error rate isn't the whole story

For a dictation product, word error rate is the metric. For a voice agent, most errors don't matter — a substitution on "the" or "um" costs nothing. The errors that break a call are on the load-bearing words: the account number, "cancel," "yes," a dollar amount. Measure task-level outcomes alongside raw accuracy: did the agent get the intent right, did it capture the right field values, did the call resolve.

How to segment

• By accent group and by language — the primary cut.
• By region or phone line, which often correlates with accent.
• By call type, since a status check tolerates more recognition error than an intake that captures a dozen fields.

You need enough calls per segment to estimate a rate you'd act on — dozens at minimum, more for a number that drives a routing decision.

The measurement method

Take a sample of real calls per segment. Transcribe them with a human reference. Compute word error rate and, more importantly, field-capture accuracy — the share of critical values the agent got exactly right. Then line those up against the operational metrics for the same segment: containment rate, escalation reasons, and satisfaction. A segment with high word error rate, low field capture, and a spike in "low confidence" escalations is one the agent isn't ready to serve.

What to do with a weak segment

• A per-segment quality gate — don't route a market to the agent until it clears a bar.
• A lower confidence threshold for that segment, so the agent escalates sooner instead of guessing.
• Keypad fallback for critical fields — let the caller enter an account number on the dialpad rather than have it misheard.
• A better model or targeted tuning for the language, if the volume justifies it.

The honest framing

Accents and languages underrepresented in a model's training data get worse recognition. This is well documented, and it's not something a vendor's blended accuracy number will show you. The only reliable read is measuring your own callers, split the way your callers actually vary.

Track it over time

Recognition accuracy isn't static. The speech model gets updated, and your caller base shifts as you enter new markets or run a campaign that pulls a different demographic. A segment that measured fine six months ago can degrade after a model change nobody flagged to you. Re-run the per-segment measurement on a schedule — quarterly is a reasonable default — and treat a model or vendor change as a trigger to re-measure before it reaches production traffic. On our voice AI engagement the per-segment containment and error numbers are part of the standing dashboard, not a one-time acceptance test.

What "good enough" looks like

There's no universal threshold, because it depends on the call type. A status-check line can tolerate a higher word error rate because the agent only needs the intent and one identifier. An intake flow that captures a dozen fields needs high field-capture accuracy or it produces bad applications. Set the bar per call type and per segment, write it down, and route anything below it to a human until it clears.

Where this stops being right

• Small segments don't have the call volume to measure reliably — combine related ones or wait for more data.
• A single-market, single-language line has less of this, though it still has within-language accent variation worth checking.
• Over-tuning to today's caller mix leaves you exposed when the mix shifts — re-measure periodically.

FAQ

Is word error rate enough? No. A voice agent cares about intent and field capture. An error on "the" is free; an error on the account number ends the call.

How many calls per segment do I need? Enough to estimate a rate with confidence — dozens minimum, more for a number you'll make routing decisions on.

What if a segment performs badly? Route it to a human, lower its escalation threshold, or add keypad fallback for critical fields. Don't ship a market where the agent fails half the calls.

ISTRALLEN builds voice agents measured per accent and language segment, not on a blended number; see AI for Fintech.