The Missing Text Layer: Why Speech-to-Speech Agents Fail Enterprise Requirements

The demo is always impressive. The voice is warm, the response comes back in under half a second, it handles interruptions like a person. Then you ask it a question that actually requires thinking, and the illusion collapses into a confident, shallow answer. Anyone who has spent more than five minutes with a voice mode knows the feeling.

The major AI providers keep racing to ship native speech-to-speech APIs anyway: one model, audio in, audio out, sealed shut. And in doing so, they are optimizing for a customer who no longer exists. The enterprises actually deploying voice agents moved past the demo stage a while ago. What they need is not a more impressive black box. It is flexibility across the speech-to-text, agent and text-to-speech chain, and a traceable record of every step in between.

At Blits.ai we build and deploy conversational AI for banks and enterprises across Europe, the Middle East and Africa. We have looked hard at native speech-to-speech models, because the naturalness really is remarkable. We keep reaching the same conclusion: for enterprise deployments, S2S is not an upgrade. It is the removal of everything that makes a voice agent deployable.

Here is why.

Enterprise voice AI is a control problem, not a latency problem

The pitch for speech-to-speech is speed and naturalness. One model, no pipeline, sub-second responses, human-like prosody. All true.

But nobody has ever cancelled an enterprise voice project because responses took 1.2 seconds instead of 0.5. Projects get cancelled because the bot said something it should not have said, because nobody could explain to a regulator why it said it, or because it could not be deployed in the required jurisdiction at all.

The classic cascaded architecture, speech-to-text into a language model into text-to-speech, is often described as legacy. Slower, more moving parts, more engineering. What that framing misses is that every one of those seams is a control point. The text layer between the stages is not overhead. It is the product.

1. No text layer means no guardrails, no redaction, no audit trail

In a cascaded pipeline, every user utterance becomes text before the model reasons about it, and every response exists as text before it is spoken. At each of those points you can intercept: mask card numbers and personal data before they reach the model, run compliance filters before the answer reaches the customer, and log a complete, human-readable record of the conversation.

When a bank's risk team asks "show us exactly what the system understood and what it decided to say", a cascaded system can answer with a transcript of every stage. A speech-to-speech model collapses all of it into one opaque audio-to-audio pass. There is no moment where the answer exists as inspectable text before it is already playing in the customer's ear.

You can transcribe the audio afterwards, but that is forensics, not control. Post-hoc transcription tells you what went wrong yesterday. It cannot stop anything today. In regulated industries, that difference is disqualifying.

2. One model, one prompt: S2S cannot do orchestration

This is the failure that surprises people who have not shipped an enterprise bot, so it is worth showing what actually sits behind one.

A production banking assistant is not a single model with a single prompt. A single customer turn typically flows through several specialized LLM calls, each with its own scoped instructions, its own model choice, and its own failure handling. A small, fast model classifies the intent: FAQ, account query, product question, escalation, off-topic. Depending on the route, the turn goes to a retrieval-augmented answer generator with knowledge base context injected, or to a natural-language-to-SQL model that carries the database schema in its prompt, or to a profiling layer that adapts tone and level of detail. A separate verification step checks the drafted answer against retrieved sources and compliance rules before it goes out. Deterministic code sits between the layers, deciding what happens next.

In one of our deployments for a major Gulf bank, this dual-LLM pattern is the entire reason the bot works: one model translates the customer's question into a structured database query, a different model turns the results into a natural answer. Each has a narrow prompt doing one job well. The combined instruction surface of a system like this runs to tens of thousands of tokens of carefully separated logic.

A speech-to-speech model gives you one model, one session, one system prompt. You cannot route a customer's turn to a different specialist model mid-conversation, because the moment you intercept between listening and speaking, you have rebuilt a cascaded pipeline and thrown away the latency advantage you paid for. Everything the system needs, routing rules, schemas, compliance constraints, knowledge, has to fit into a single prompt. On Azure's Realtime API, that prompt lives within a 32,000 token input ceiling, roughly a quarter of what the same vendor's text models accept.

And the model interpreting that prompt follows instructions worse than its text counterpart. The vendors say so themselves, indirectly: each new realtime model generation is marketed on "improved instruction following", which tells you exactly where the previous ones were failing. The reason is structural. Instruction-following depth comes from years of text-based post-training, and audio-native models have had far less of it. The instruction layer of S2S is not shallow by accident. It is shallow because the modality is young.

3. The intelligence ceiling: great at small talk, lost on real questions

Anyone who has used a voice mode for more than a few minutes knows this one from experience. Ask about the weather, get a perfect answer. Ask a question that requires actual reasoning, comparing two loan products against your situation, walking through a multi-step process, reconciling conflicting information, and the answer turns confident, shallow and often wrong.

This is not a temporary quality gap that the next model version fixes. It is built into the deal S2S offers. The whole value proposition is sub-second response, which means two things. First, the models behind realtime APIs are the small, fast variants of each vendor's lineup, not the frontier models, because nobody can serve a frontier model at conversational latency and conversational cost. Second, and more fundamentally: the biggest capability jump in language models over the past two years came from letting models think before answering, spending seconds of deliberate reasoning on hard problems. A realtime voice loop has no seconds to spend. S2S is structurally locked out of exactly the mechanism that made models smart.

A cascaded architecture does not have to make this trade for every turn. The orchestration layer sees the question as text and can route by difficulty: simple intents get an instant answer from a fast model, complex questions get escalated to a stronger model, with the voice layer buying the thinking time naturally ("let me check that for you"). Humans on the phone do exactly this. The pause before a considered answer is not a defect, it is a signal of quality. S2S cannot make that choice, because there is no point in the loop where anything can be routed anywhere.

So the honest positioning of today's S2S agents is narrow: simple questions, simple commands, low stakes. And even those easy tasks run ungoverned, so you have no way to know how well they are actually being handled. In consumer products, nobody measures and nobody minds. In an enterprise, "it seems to work" is not a quality metric, and it is certainly not an answer for a customer asking why a transaction was declined.

4. You cannot deploy it where your clients are

For our clients in the Gulf, data residency is not a preference. In Saudi Arabia, banks must store and process customer data on infrastructure inside the Kingdom, and hosting financial data offshore requires prior approval from the central bank. Similar expectations apply across the region. Audio of a customer discussing their finances is about as sensitive as data gets.

Now check the deployment map for the leading S2S offerings. As of today, Azure's GPT Realtime models are available in exactly two regions worldwide, East US 2 and Sweden Central, and only as global deployments, which means prompts and responses may be processed in any Azure region where the model runs. Azure's Data Zone deployments, the mechanism that pins processing to a geography, exist for the United States, the European Union and Asia Pacific. There is no Middle East data zone. The picture at Google is the same: Vertex AI's residency commitments for model processing cover US and EU jurisdictions, its global endpoints explicitly provide no residency guarantee, and the newest live-audio models have shipped global-first, with regional availability trailing behind.

A cascaded pipeline does not have this problem. Speech recognition and synthesis can run in-region, where the platform guarantees processing stays inside the region of the resource, or entirely on-premises in containers inside the client's own infrastructure. The language model in the middle can be a regionally deployed service or a self-hosted open-weight model behind the client's firewall. With S2S, in-region deployment is not a configuration you can request. It is architecturally unavailable.

5. You cannot regression-test what you cannot read

Enterprises do not ship what they cannot test. And here is what testing actually looks like when correctness is the product. Before any change goes live on our platform, prompts, knowledge bases and routing logic run against evaluation sets: hundreds of known questions with expected behaviors, diffed against the previous version. Every failure gets classified, wrong retrieval, wrong grounding, wrong format, wrong language, so we fix causes, not symptoms. We evaluate transcription quality per dialect. We benchmark speech engines on our own infrastructure instead of trusting datasheets. It is unglamorous work, and it is most of the work. Correct is not a property a voice agent has. It is a process someone runs, every release, forever.

That entire discipline assumes a text layer. With a cascaded system you can evaluate each stage in isolation: is transcription accurate on this dialect, does retrieval surface the right documents, does the answer stay grounded in them. When something breaks in production, the transcript tells you which stage failed.

With speech-to-speech, evaluation means generating audio, running it through the model, transcribing the output and hoping your measurement pipeline is not introducing more noise than the change you are trying to detect. When a customer complains that the bot said something strange, there is no intermediate state to inspect. Debugging becomes guessing.

6. The language reality nobody demos

The demos are in English. Our deployments are in Gulf Arabic, Egyptian Arabic, Saudi dialect, and customers who switch between Arabic and English inside a single sentence.

In a cascaded pipeline, dialect handling is hard but tractable, precisely because each stage can be measured and fixed independently. We can evaluate transcription quality per dialect, tune retrieval for dialect variants, and control whether the bot answers in Modern Standard Arabic or the customer's own dialect. When quality drops, we know which stage to patch.

With speech-to-speech, you get whatever the model learned. If its Saudi dialect handling is weak, there is no stage to swap out, no component to fine-tune, no knob to turn. You file a ticket and wait.

What S2S gets right

To be fair to the technology: the naturalness gap is real. Native speech-to-speech models handle interruptions, prosody, emotion and conversational timing in ways a cascaded pipeline still cannot fully match. A pipeline can sound flat where S2S sounds alive. For consumer companions, language practice, entertainment, that wins.

But enterprises are not buying naturalness. They are buying correct answers, delivered safely, in the right jurisdiction, with a paper trail.

The alternative, and how fast it actually is

Our answer is a streaming cascaded architecture, and the latency story is better than most people assume. The mental model most people carry is sequential: transcribe the whole utterance, then generate the whole answer, then synthesize the whole audio. Add those up and you get the sluggish voice bots everyone remembers.

A streaming pipeline overlaps everything. Speech-to-text emits partial transcripts while the customer is still talking, so by the moment they stop, the orchestration layer has already routed the intent and started retrieval. The language model streams its answer token by token, and text-to-speech starts speaking the first sentence while the rest is still being generated. No stage waits for the previous one to finish.

The numbers deserve honesty, because practitioners will check them. Published benchmarks show what the fastest stacks achieve: streaming speech recognition in 100 to 300 milliseconds, first tokens from a well-chosen language model in 300 to 500, first synthesized audio in under 200. Overlapped and co-located, that lands voice-to-voice response in the 600 to 800 millisecond range.

Real enterprise deployments rarely get to pick that stack. Compliance-approved vendors, regional hosting and non-English languages all cost speed. The fastest engines are English-first. Several vendors now advertise fast Arabic streaming, but those figures measure time-to-first-partial under ideal conditions, and dialect accuracy claims made on vendor-chosen test sets rarely survive contact with real call audio. Whether a fast engine's Arabic is actually usable is a question only your own evaluation framework can answer. In practice, a production-grade multilingual pipeline lands somewhere between one and a half and two seconds. Native S2S responds in roughly 200 to 300 milliseconds.

So yes, the gap is real. Here is why it does not decide anything: in human conversation, the average pause between speakers is around 200 milliseconds, but people tolerate far longer pauses when the pause carries meaning. A human agent who asks for a moment and then answers correctly feels more competent, not less. At one and a half to two seconds, you do not leave that pause empty: an instant acknowledgment lands within a conversational beat, resets the customer's clock, and buys the pipeline its thinking time. A considered answer after two seconds beats a wrong answer after 250 milliseconds. Every single time. Enterprises are not losing customers to two seconds of latency. They are losing them to wrong answers, and to voice projects that never clear compliance at all.

What we learned making cascaded fast

Getting a pipeline into that range took real engineering, and the lessons were not the ones we expected going in.

Most of the latency lives in the seams, not the models. Our biggest early wins came not from swapping models but from architecture hygiene: every network hop between separately hosted STT, LLM and TTS services adds tens of milliseconds, and a cascaded turn involves many hops. Co-locating the services in one region, keeping connections warm and cutting serialization overhead recovered more time than any model upgrade.

Endpointing is the hidden giant. Deciding that the customer has finished speaking is its own latency source, and a naive silence threshold quietly burns 500 milliseconds or more before the pipeline even starts. Smarter turn detection, deciding the customer finished a thought rather than merely paused, combined with the ability to gracefully abort if they resume, wins back more time than faster synthesis ever will.

Vendor latency claims are a class system, and you have to benchmark it yourself. When we benchmarked text-to-speech engines on our own infrastructure, the spread was not percentages, it was multiples: the newer LLM-based synthesis models measured roughly 600 to 1,100 milliseconds to first audio in streaming mode depending on the model, and four to seven seconds in full-synthesis mode, while purpose-built streaming engines advertise under 200. Same product category, a tenfold difference. But milliseconds are only half the benchmark. The speed comes at a price: the fastest engines trade away voice quality, and the trade gets worse the further you move from English. Prosody flattens, pronunciation slips, and in Arabic the difference between a natural voice and a robotic one is immediately audible to every customer. That is why we sometimes run a slower engine on purpose. A bank's voice is part of its brand, and 500 saved milliseconds do not compensate for a voice that sounds cheap. Two more lessons came out of that. Engine class beats engine tuning: no configuration flag turns a quality-first synthesis model into a latency-first one. And integration mode can silently cost more than the engine itself: call a streaming-capable engine in full-synthesis mode, or hold the LLM's complete answer before sending the first sentence to synthesis, and you pay seconds for a sub-second engine.

Perceived latency is not measured latency. An immediate, natural acknowledgment makes a longer computation feel instant. This is also where the orchestration advantage compounds: because the router sees the question as text, it knows within the first couple hundred milliseconds whether this is a simple turn to answer instantly or a hard one worth escalating to a stronger model behind a brief acknowledgment.

Speed is a budget, not a score. Every millisecond spent in one stage is unavailable to the others. Treating the pipeline as a strict end-to-end budget, and measuring every stage in production rather than trusting vendor numbers, is the discipline that keeps it honest. It is also, not coincidentally, exactly the kind of stage-by-stage measurement that S2S makes impossible.

The market does not sit still, so the platform cannot either. New speech-to-text and text-to-speech models ship constantly, and the leaderboard reshuffles every few months. We run continuous tests on both stages, and because every stage of our platform is independently measurable and swappable, adopting a better engine is routine: when a new model arrives, we benchmark it against our evaluation sets, verify it actually wins for the specific use case, the language, the dialect, the voice, and put it into production. The whole cycle takes days, not months. That is the compounding advantage of a diagnosable architecture: every improvement anywhere in the market becomes our improvement, on our schedule. With a sealed speech-to-speech box, the model is the product. You get better only when the vendor does, all at once, with no way to verify what else changed.

Where this ends up

Speech-to-speech will earn its place in the enterprise eventually. Not like this. The architecture that wins is already visible: a speech-native front-end handling what S2S is genuinely good at, turn-taking, prosody, interruptions, wrapped around a text-native orchestration brain that keeps every control point, every specialist model, every audit trail. The mouth can be speech-native. The brain has to think in text.

The first provider to expose an interceptable text layer inside the realtime loop, without giving up the latency, wins the enterprise voice market. Until then, choosing S2S for enterprise voice means choosing the demo over the deployment.

We know which one our clients are paying for.