How to Score a TTS Bake-Off When the Demo Is Lying to You

A TTS demo always sounds perfect, and that alone should make you careful. A vendor plays a thirty second clip of a warm, unhurried voice reading a script that someone tuned by hand the night before. Of course it sounds human. Put the same model behind a live banking line at 4 p.m. on a Tuesday, reading an account number a caller has just rattled off in a mix of Arabic and English, and it fails in ways the demo could never show you.

If your RFP still opens with a MOS score, you are grading the one property that no longer separates the vendors.

Naturalness no longer separates vendors

By 2026 naturalness is largely a solved problem, and the leaderboards show it. On the Artificial Analysis Speech Arena, which ranks models by Elo from blind human A/B votes, the top of the table is crowded: Qwen-Audio-3.0-TTS-Plus at 1236, Simba 3.2 at 1234, Gemini 3.1 Flash TTS at 1214 and Cartesia Sonic 3.5 at 1207. The field's best models sit within roughly 30 Elo points of each other. An open model with 82 million parameters, Kokoro-82M, topped a naturalness arena against rivals five to fourteen times its size. You can now reach the naturalness ceiling with a model that runs offline on a laptop.

MOS itself, the mean opinion score panel that still anchors most procurement decks, becomes worse than uninformative once systems are this good. Academic work through 2025 keeps making the same point. MOS has limited resolution as models approach human quality, raters disagree widely, and scores move depending on whether raters hear isolated sentences or full paragraphs. Claims of human parity "often fail under deception testing." A panel is being asked to split hairs that are already indistinguishable.

So do not lead with it. Treat naturalness as a minimum requirement and grade the things that still vary by an order of magnitude.

What still varies by a factor of ten

There are mainly three such things: how fast the first audio arrives under real load, whether the model reads your strings correctly, and whether it holds up in a live back and forth. A demo shows none of them, because a demo is a single request on a clean script from a machine in the same data center, with nobody talking back.

Latency comes first, because almost everyone believes they have measured it and very few have.

Advertised latency is a marketing figure: best case, single request, same data center. Measured latency is what the caller actually experiences, and the two can be far apart. When Coval ran an independent probe of production endpoints on May 4, 2026 (surfaced via Gradium), ElevenLabs Flash v2.5 recorded a P50 time to first audio of 288ms, against a marketed figure near 135ms. Cartesia Sonic-3 was the best of the major streaming models at 188ms. OpenAI's batch TTS-1-HD landed at 2,295ms, more than two seconds before the first sound. That last result says as much about which endpoint you wired up as about the vendor.

The average also hides the number that does the most damage.

The mean latency tells you how the call sounds in the brochure. The P95 and the jitter tell you how it sounds at 4 p.m.

Averages smooth over the tail. A model with a decent mean and a 380ms interquartile range (Coval's measurement for Rime Mist-v3) will still produce calls that feel broken, because the occasional slow turn is what makes a caller start talking over the bot. Report P50 and P95, and report the spread. A steady 250ms is better than a 150ms mean with a long tail.

Building the test set from your own traffic

Teams often skip this step, and it determines most of the outcome. Vendors benchmark their models on clean text. Your production text contains 1/4, $5.7M, PRM423GDDML2354, a drug name no phonetic dictionary has seen, and dates in whatever format your CRM happens to store. Text normalization is the most common production failure mode in TTS, and it stays invisible until something goes wrong. Without context, 1/4 could be "January the fourth," "April first," "one fourth" or "one slash four." A model that wins the naturalness arena can still read your customer's balance wrongly, confidently and in a beautiful voice.

Build the test set from a corpus of your real utterances rather than from benchmark boilerplate:

• Pull a few hundred actual turns from your support and sales transcripts, choosing the messy ones over the tidy ones.
• Oversample the entities that carry risk: account numbers, currency amounts, IBANs, dates, reference codes, product SKUs, medication and policy names.
• Include the code switching you actually see. If half your callers move between Gulf Arabic and English mid sentence, your test set has to do the same.
• Add the noise of a real call, such as mobile microphones and background chatter, instead of studio conditions.

A representative dataset will sound a little embarrassing, because real customers do. The purpose is to make the model fail on the same inputs your customers will give it.

The measurements that belong in the protocol

Once the corpus exists, scoring is mechanical. A handful of checks do most of the work. Each gets a threshold that you set before listening, so that you grade against a fixed bar instead of talking yourself into whichever voice you liked best.

Round trip intelligibility. The cheapest automatic proxy for "did it say the entity correctly" is to run the synthesized audio back through a good speech to text engine and compare the transcript with the source text. It will not catch every prosody problem, but it does catch the ones that matter for compliance. A mangled account number or a dropped decimal shows up immediately as a word error. On Coval's run, word error rates clustered low (Flash v2.5 at 5.2%, Aura-2 at 6.4%), but that was on generic text. On your entities the spread widens quickly.

Entity and normalization accuracy on your strings. Score this separately and strictly, because it is the failure that destroys trust on the first high stakes transaction. Some vendors have designed for exactly this case. Rime's Mist line is deterministic and IVR focused, with a spell() control for reading codes letter by letter and strong number and currency handling. Deepgram claims Aura-2 reaches 89% "good" accuracy on enterprise edge cases, against 53 to 75% for rivals. That figure comes from Deepgram's own marketing, so treat it as a hypothesis to test and not as a fact to quote. In any case, the figure that counts is the one you measure on your own account numbers.

Barge in and turn taking. A voice agent stands or falls on how it handles interruption. The 2026 production bar is now quantified: barge in detection under ~400ms from speech onset, a false barge in rate under 2%, missed true interruptions under 1%, TTS flush under 60ms, and a total round trip of roughly 800ms for the conversation to feel natural. These are pipeline figures, not TTS figures, and a 90ms TTS behind a slow LLM gains you nothing. You are benchmarking the whole stack (speech to text, reasoning, synthesis and the barge in flush together). That is why serious investment is moving to simulation in the style of Coval, which replays thousands of noisy callers on every model swap, a method borrowed from how autonomous vehicle teams test.

Determinism. In regulated voice, expressiveness is often a liability. A bank or telco needs the same balance, the same drug name and the same confirmation code pronounced identically on every call, because auditability requires reproducibility. Here the flagship expressive models tend to rule themselves out. ElevenLabs released v3 to general availability in March 2026 with bracketed audio tags and a claimed 68% reduction in complex text errors, and explicitly excluded it from real time use, because "there is no way to get v3 quality at Flash speeds." A model that improvises emotion is the wrong tool when a compliance officer needs last month's call to sound like this month's.

Voice cloning belongs on the scorecard

The three second voice clone that impresses in a demo becomes an active attack surface as soon as you deploy voice at scale.

The FBI logged more than 22,000 reports of AI voice and video scams in 2025, with reported losses near $893M. In one case a CFO authorized a $243,000 transfer after a call that convincingly mimicked the CEO, using a cloned deepfake voice. "Can it clone a voice from a short clip" belongs in your threat model as a red flag, not in a list of features. For that reason watermarking now belongs on the scorecard as well. Every output from Gemini 3.1 Flash TTS is watermarked with SynthID by default. Put consent controls, liveness checks and provenance on the same page as latency. Otherwise you may select a model that sounds excellent and opens a security gap.

Keep the humans, add the machines

None of this replaces a listening panel. It changes what the panel is for.

Automation tells you whether you kept quality after the last deploy. A round trip WER check, an objective metric based on distributions such as TTSDS2, and a latency probe running in CI will catch regressions as soon as a vendor ships a new endpoint. (TTSDS2 was the only one of sixteen tested metrics to clear a Spearman correlation of 0.50 in every domain, which makes it a credible alternative to paying for MOS panels.) Native speaking testers tell you what automation cannot: whether a Gulf Arabic speaker accepts this voice as legitimate for a bank, and the "almost right" failures that damage trust and never show up in a spectrogram. We cover why that judgment is so hard to automate in our piece on dialects and the voice quality gap. This article is its companion and deals with how to measure it.

Humans define what "good" means in a given culture, and machines check whether you have held on to it. You need both, and you need them running continuously, because agentic systems change often and a choice made once goes stale.

A scorecard that grades what matters looks roughly like this:

TTS bake-off scorecard (weight to your risk, not the vendor's demo)

Naturalness (blind pairwise, your listeners) ....... pass/fail gate, not a tiebreaker
P50 time-to-first-audio, your region, under load ... target and measured
P95 time-to-first-audio + jitter (IQR) ............. the tail is the number that matters
Round-trip intelligibility (TTS -> STT -> diff) .... WER on YOUR entities, not generic text
Entity/normalization accuracy ...................... % correct on account #s, $, dates, drug names
Barge-in detection / flush / round-trip ............ <400ms / <60ms / ~800ms, end-to-end
Determinism (same string, same output) ............. required for regulated voice
Voice-security posture ............................. watermarking, consent, liveness, cloning risk

Weight the rows to your own situation. An IVR that reads balances all day should weight determinism and entity accuracy above expressiveness, while a sales assistant weights latency and warmth more heavily. There is no universal winner, which is why the provider market is a menu and not a ranking.

A benchmark built from your own calls

The best sounding model available can still be the wrong choice, and a leaderboard will not tell you so, because leaderboards grade naturalness and every leading vendor already does that well.

A useful benchmark is built from your prompts, your noise profile, your entities and your definition of acceptable. It measures first audio under load, reports the tail as well as the mean, checks that the model reads your account numbers the same way twice, and treats voice cloning as a risk rather than a selling point. That is less striking than a slide with one large number on it. It is also what distinguishes a voice that impresses a steering committee from one that holds up on a Tuesday afternoon in production. If you want to discuss specific thresholds and tie them back to KPIs that mean something, that is a conversation we are glad to have. The most useful starting material is your worst transcripts.

Updated on 12 October 2026.