The demo is always impressive. The voice is warm, the response comes back in under half a second, and it handles interruptions like a person would. Then you ask something that requires actual thinking and get a confident, shallow answer. Anyone who has spent more than five minutes with a voice mode knows the feeling.
The major AI providers keep racing to ship native speech to speech APIs anyway: one model, audio in, audio out, sealed shut. In doing so they are optimizing for a customer who no longer exists. The enterprises that deploy voice agents moved past the demo stage a while ago. They need flexibility across the chain of speech to text, agent and text to speech, plus a traceable record of every step in between, and a more impressive black box gives them neither.
At Blits.ai we build and deploy conversational AI for banks and enterprises across Europe, the Middle East and Africa. We have looked hard at native speech to speech models, because the naturalness really is remarkable. We keep reaching the same conclusion, though. For enterprise deployments, S2S removes everything that makes a voice agent deployable, and the rest of this post explains why we think so.
The pitch for speech to speech is speed and naturalness: one model, no pipeline, responses in under a second, prosody that sounds human. All true.
But nobody has ever cancelled an enterprise voice project because responses took 1.2 seconds instead of 0.5. Projects get cancelled because the bot said something it should not have said, because nobody could explain to a regulator why it said it, or because it could not be deployed in the required jurisdiction at all.
The classic cascaded architecture (speech to text into a language model into text to speech) is often described as legacy, with more latency, more moving parts and more engineering. That framing misses that every one of those seams is a control point. We treat the text layer between the stages as the product itself.
In a cascaded pipeline, every user utterance becomes text before the model reasons about it, and every response exists as text before it is spoken. At each of those points you can intercept. You can mask card numbers and personal data before they reach the model, run compliance filters before the answer reaches the customer, and log a complete record of the conversation that a person can read.
When a bank's risk team asks "show us exactly what the system understood and what it decided to say", a cascaded system can answer with a transcript of every stage. A speech to speech model collapses all of it into one opaque pass from audio to audio. At no moment does the answer exist as inspectable text before it is already playing in the customer's ear.
You can transcribe the audio afterwards, but that is forensics. Transcription after the fact tells you what went wrong yesterday and cannot stop anything today, and in regulated industries that difference is disqualifying.
This failure surprises people who have not shipped an enterprise bot, so it is worth showing what sits behind one.
A production banking assistant is not a single model with a single prompt. One customer turn typically flows through several specialized LLM calls, each with its own scoped instructions, its own model choice and its own failure handling. A small, fast model classifies the intent (FAQ, account query, product question, escalation, off topic). Depending on the route, the turn goes to a retrieval augmented answer generator with knowledge base context injected, to a model that translates natural language to SQL and carries the database schema in its prompt, or to a profiling layer that adapts tone and level of detail. A separate verification step checks the drafted answer against retrieved sources and compliance rules before it goes out, and deterministic code between the layers decides what happens next.
In one of our deployments for a major Gulf bank, this dual LLM pattern is the entire reason the bot works. One model translates the customer's question into a structured database query and a different model turns the results into a natural answer, each with a narrow prompt doing one job well. The combined instruction surface of a system like this runs to tens of thousands of tokens of carefully separated logic.
A speech to speech model gives you one model, one session and one system prompt. You cannot route a customer's turn to a different specialist model halfway through, because the moment you intercept between listening and speaking you have rebuilt a cascaded pipeline and thrown away the latency advantage you paid for. Everything the system needs (routing rules, schemas, compliance constraints, knowledge) has to fit into a single prompt. On Azure's Realtime API that prompt lives within a 32,000 token input ceiling, roughly a quarter of what the same vendor's text models accept.
The model interpreting that prompt also follows instructions worse than its text counterpart. The vendors admit as much, indirectly: each new realtime model generation is marketed on "improved instruction following", which tells you where the previous ones were failing. The cause is structural. Depth in following instructions comes from years of post training on text, and audio native models have had far less of it, so the instruction layer of S2S stays shallow for as long as the modality is young.
Anyone who has used a voice mode for more than a few minutes knows this from experience. Ask about the weather and you get a perfect answer. Ask something that requires reasoning, such as comparing two loan products against your situation, walking through a process with several steps or reconciling conflicting information, and the answer turns confident, shallow and often wrong.
The next model version will not fix this, because it is built into the deal S2S offers. The whole value proposition is a response in under a second, and that has two consequences. First, the models behind realtime APIs are the small, fast variants of each vendor's lineup and not the frontier models, since nobody can serve a frontier model at conversational latency and conversational cost. Second, and more fundamentally, the biggest capability jump in language models over the past two years came from letting models think before answering, spending seconds of deliberate reasoning on hard problems. A realtime voice loop has no seconds to spend, which locks S2S out of exactly the mechanism that made models smart.
A cascaded architecture does not have to make this trade on every turn. The orchestration layer sees the question as text and can route by difficulty. Simple intents get an instant answer from a fast model, and complex questions go to a stronger model while the voice layer buys the thinking time naturally ("let me check that for you"). Humans on the phone do exactly this, and a short pause before a considered answer usually reads as care. S2S cannot make that choice, because there is no point in the loop where anything can be routed anywhere.
So a fair positioning of today's S2S agents is narrow: simple questions, simple commands, low stakes. Even those easy tasks run ungoverned, so you have no way to know how well they are being handled. In consumer products nobody measures and nobody minds. In an enterprise, "it seems to work" is not a quality metric, and it is certainly not an answer for a customer asking why a transaction was declined.
For our clients in the Gulf, data residency is a legal requirement. In Saudi Arabia, banks must store and process customer data on infrastructure inside the Kingdom, and hosting financial data offshore requires prior approval from the central bank. Similar expectations apply across the region, and audio of a customer discussing their finances is about as sensitive as data gets.
Now check the deployment map for the leading S2S offerings. As of today, Azure's GPT Realtime models are available in exactly two regions worldwide, East US 2 and Sweden Central, and only as global deployments, which means prompts and responses may be processed in any Azure region where the model runs. Azure's Data Zone deployments, the mechanism that pins processing to a geography, exist for the United States, the European Union and Asia Pacific. There is no Middle East data zone. Google looks the same. Vertex AI's residency commitments for model processing cover US and EU jurisdictions, its global endpoints explicitly provide no residency guarantee, and the newest live audio models shipped globally first, with regional availability trailing behind.
A cascaded pipeline does not have this problem. Speech recognition and synthesis can run in region, where the platform guarantees processing stays inside the region of the resource, or entirely on premises in containers inside the client's own infrastructure. The language model in the middle can be a regionally deployed service or a self hosted open weight model behind the client's firewall. With S2S you cannot request deployment in region as a configuration option, because the architecture does not offer it.
Enterprises do not ship what they cannot test, and when correctness is the product, testing looks like this. Before any change goes live on our platform, prompts, knowledge bases and routing logic run against evaluation sets: hundreds of known questions with expected behaviors, diffed against the previous version. Every failure gets classified (wrong retrieval, wrong grounding, wrong format, wrong language) so we fix causes instead of symptoms. We evaluate transcription quality per dialect, and we benchmark speech engines on our own infrastructure instead of trusting datasheets. It is slow, repetitive work, and it is most of the work. A voice agent stays correct only because someone runs that process on every release, indefinitely.
That entire discipline assumes a text layer. With a cascaded system you can evaluate each stage in isolation. Is transcription accurate on this dialect, does retrieval surface the right documents, does the answer stay grounded in them? When something breaks in production, the transcript tells you which stage failed.
With speech to speech, evaluation means generating audio, running it through the model, transcribing the output and hoping your measurement pipeline adds less noise than the change you are trying to detect. When a customer complains that the bot said something strange, there is no intermediate state to inspect, and debugging turns into guessing.
Language makes all of this harder. The demos are in English. Our deployments are in Gulf Arabic, Egyptian Arabic and Saudi dialect, with customers who switch between Arabic and English inside a single sentence. In a cascaded pipeline, dialect handling is hard but tractable, precisely because each stage can be measured and fixed on its own. We can evaluate transcription per dialect, tune retrieval for dialect variants, and control whether the bot answers in Modern Standard Arabic or the customer's own dialect, and when quality drops we know which stage to patch. With speech to speech you get whatever the model learned. If its Saudi dialect handling is weak, there is no stage to swap out, no component to fine tune and no knob to turn. You file a ticket and wait.
The naturalness gap is real, and S2S deserves credit for it. Native speech to speech models handle interruptions, prosody, emotion and conversational timing in ways a cascaded pipeline still cannot fully match, and a pipeline can sound flat where S2S sounds alive. For consumer companions, language practice and entertainment, that wins.
Enterprises, though, are paying for correct answers, delivered safely, in the right jurisdiction, with a paper trail. Naturalness comes a long way down that list.
Our answer is a streaming cascaded architecture, and the latency story is better than most people assume. The mental model most people carry is sequential: transcribe the whole utterance, then generate the whole answer, then synthesize the whole audio. Add those up and you get the sluggish voice bots everyone remembers.
A streaming pipeline overlaps everything. Speech to text emits partial transcripts while the customer is still talking, so by the time they stop, the orchestration layer has already routed the intent and started retrieval. The language model streams its answer token by token, and text to speech starts speaking the first sentence while the rest is still being generated. No stage waits for the previous one to finish.
Practitioners will check the numbers, so here they are as published. Benchmarks show what the fastest stacks achieve: streaming speech recognition in 100 to 300 milliseconds, first tokens from a well chosen language model in 300 to 500, first synthesized audio in under 200. Overlapped and running in the same location, that puts the response from voice to voice in the 600 to 800 millisecond range.
Real enterprise deployments rarely get to pick that stack. Compliance approved vendors, regional hosting and languages other than English all cost speed, and the fastest engines are built for English first. Several vendors now advertise fast Arabic streaming, but those figures measure time to first partial under ideal conditions, and dialect accuracy claims made on test sets the vendor picked rarely survive contact with real call audio. Only your own evaluation framework can tell you whether a fast engine's Arabic is usable. A multilingual pipeline in production lands somewhere between one and a half and two seconds, while native S2S responds in roughly 200 to 300 milliseconds.
So the gap is real, but it does not decide anything. In human conversation the average pause between speakers is around 200 milliseconds, yet people tolerate far longer pauses when the pause carries meaning, and a human agent who asks for a moment and then answers correctly comes across as more competent. At one and a half to two seconds you do not leave that pause empty. An instant acknowledgment lands within a conversational beat, resets the customer's clock and buys the pipeline its thinking time. We will take a considered answer after two seconds over a wrong one after 250 milliseconds every time. Enterprises are losing customers to wrong answers, and to voice projects that never clear compliance at all, far more than to two seconds of latency.
Getting a pipeline into that range took real engineering, and the lessons were not the ones we expected going in.
Most of the latency lives in the seams between models. Our biggest early wins came from architecture hygiene rather than from swapping models. Every network hop between separately hosted STT, LLM and TTS services adds tens of milliseconds, and a cascaded turn involves many hops. Putting the services in one region, keeping connections warm and cutting serialization overhead recovered more time than any model upgrade.
Endpointing is the hidden giant. Deciding that the customer has finished speaking is its own latency source, and a naive silence threshold burns 500 milliseconds or more before the pipeline even starts, without anyone noticing. Smarter turn detection (deciding the customer finished a thought rather than merely paused), combined with the ability to abort gracefully if they resume, wins back more time than faster synthesis ever will.
Vendor latency claims form a class system, and you have to benchmark it yourself. When we benchmarked text to speech engines on our own infrastructure, the spread came in multiples rather than percentages. The newer synthesis models built on LLMs measured roughly 600 to 1,100 milliseconds to first audio in streaming mode depending on the model, and four to seven seconds in full synthesis mode, while engines built specifically for streaming advertise under 200. Same product category, a tenfold difference.
Milliseconds are only half the benchmark, though. The fastest engines trade away voice quality, and the trade gets worse the further you move from English. Prosody flattens, pronunciation slips, and in Arabic the difference between a natural voice and a robotic one is immediately audible to every customer. That is why we sometimes run a slower engine on purpose. A bank's voice is part of its brand, and 500 saved milliseconds do not make up for a voice that sounds cheap. Two more lessons came out of that work. Engine class beats engine tuning, since no configuration flag turns a synthesis model designed for quality into one designed for latency. And integration mode can silently cost more than the engine itself. Call an engine that supports streaming in full synthesis mode, or hold the LLM's complete answer before sending the first sentence to synthesis, and you pay seconds for an engine that could answer in well under one.
Perceived latency differs from measured latency. An immediate, natural acknowledgment makes a longer computation feel instant. This is also where the orchestration advantage compounds. Because the router sees the question as text, it knows within the first couple hundred milliseconds whether this is a simple turn to answer instantly or a hard one worth escalating to a stronger model behind a brief acknowledgment.
Treat speed as a budget. Every millisecond spent in one stage is unavailable to the others. Treating the pipeline as a strict budget from end to end, and measuring every stage in production rather than trusting vendor numbers, keeps it fast over time. That kind of measurement, stage by stage, is also exactly what S2S makes impossible.
The market does not sit still, so the platform cannot either. New speech to text and text to speech models ship constantly, and the leaderboard reshuffles every few months. We run continuous tests on both stages, and because every stage of our platform can be measured and swapped independently, adopting a better engine is routine. When a new model arrives we benchmark it against our evaluation sets, check that it wins for the specific use case (the language, the dialect, the voice) and put it into production, a cycle that takes days. Every improvement anywhere in the market becomes our improvement, on our schedule. With a sealed speech to speech box the model is the product, and you get better only when the vendor does, all at once, with no way to verify what else changed.
Speech to speech will earn its place in the enterprise eventually, in a different form from today's. The architecture that wins is already visible. A front end built for speech handles what S2S is good at (turn taking, prosody, interruptions), wrapped around an orchestration brain that works in text and keeps every control point, every specialist model and every audit trail. The mouth can work natively in speech, as long as the brain thinks in text.
The first provider to expose an interceptable text layer inside the realtime loop, without giving up the latency, wins the enterprise voice market. Until then, choosing S2S for enterprise voice means choosing the demo over the deployment, and we know which of the two our clients are paying for.
Updated on 12 October 2026.
