A customer calls her bank to report a lost card. The voice agent answers at once, understands her, and blocks the card in under a minute. Another customer calls the same line. He speaks a regional dialect, there's traffic in the background, and after every sentence there is a pause of almost two seconds. He hangs up and calls again, hoping for a person.
Same voice agent, same model, completely different experience. An AI voice agent is software that holds a spoken conversation and gets things done: it listens, understands, acts in your systems and speaks the answer. In enterprise settings, success depends less on how clever it is and more on three things a demo rarely shows: speed, language and compliance.
In a chat, a two second wait feels normal. On the phone, it feels broken. Research on conversation across languages shows how precise human turn taking is: gaps between speaking turns average around just 200 milliseconds, and a study across ten languages found a general avoidance of overlapping talk and a minimization of silence between turns in all of them.
A voice agent can't match 200 milliseconds every time, but every part of the chain eats into the budget. Speech has to be recognized, the model has to decide what to do, it may call a system to check a balance, and the answer has to be turned into speech again. Each step adds delay, and the customer hears all of it as one silence.
What we do to keep that silence short is mostly plain engineering. Speech is streamed instead of waiting for the end of a sentence. The answer is spoken as soon as its first sentence is ready, while the rest is still being generated. Short confirmations get a fast path. And we measure latency end to end, on real phone lines, not in a quiet office.
Some vendors promise to remove the problem with speech to speech models that skip the text step. They are fast and sound natural, but they remove the layer where guardrails, redaction and audit trails live. I explained why that trade off rarely works for regulated organizations in The Missing Text Layer.
The second problem is quieter and more serious. Speech recognition works best for the voices it was trained on. A Stanford led study tested five commercial speech recognition systems, from Amazon, Apple, Google, IBM and Microsoft, and found an average word error rate of 35 percent for black speakers, against 19 percent for white speakers.
The same pattern shows up with dialects. Arabic speech recognition faces unique challenges due to diglossia, extensive regional dialect variation, and limited speech resources, as a recent dialect dataset paper puts it. A system that performs well on Modern Standard Arabic can struggle with Gulf, Egyptian or Levantine speech, and the customer experiences that as not being understood.
For an enterprise, this is not only a quality issue. If your voice agent understands some customer groups much worse than others, those customers get a worse service from the same company. That is why we test every candidate speech model on recordings from the actual market, with native speakers listening alongside the automatic scores. We publish what we learn about Arabic dialects and wrote about it in Agentic AI Languages and Dialects.
If your voice agent understands some customers much worse than others, those customers get a worse company.
Tell callers they're talking to AI. Under Article 50 of the EU AI Act, people must be informed that they are interacting with an AI system, and that obligation applies since 2 August 2026. On a phone call, that means saying it at the start, briefly and clearly.
Outbound calls need consent. In the United States, the FCC ruled unanimously in February 2024 that AI generated voices are "artificial" under the Telephone Consumer Protection Act, which means the consent rules for prerecorded calls apply to AI voice agents too. Other markets have their own rules on telemarketing and consent. Outbound voice campaigns need a legal check per country, every time.
Never use voice alone to authenticate. Voice cloning has made a voice a weak proof of identity. The US Treasury's financial crimes unit FinCEN has warned financial institutions about deepfake media, including schemes where criminals impersonate executives or family members. Starling Bank warned that the technology can replicate a person's voice from as little as three seconds of audio. A voice agent that can block a card or change an address should confirm identity through a second channel, such as the banking app or a one time code.
Personal data in audio needs care. Cloud speech recognition hears the audio before any text exists, and cloud text to speech hears every number it reads out. For sensitive environments, ask customers to type account or card numbers instead of saying them, read back only the last digits, or run speech recognition and synthesis on your own servers.
- How fast does it respond, measured end to end on our phone lines? Ask for the distribution, not the best case.
- How well does it understand our customers? Test it on your own recordings, in your customers' dialects, with background noise.
- What happens when it isn't sure? It should confirm numbers and names, and ask again rather than guess.
- How does it hand over to a person, and does the agent see the full conversation?
- How does it handle disclosure, consent and authentication in each market where you'll use it?
- Where do guardrails and logging live, and can you read back exactly what was said and done on every call?
A voice agent that passes these six questions can take a real share of your calls, and your customers will mostly notice that they didn't have to wait. We build and test voice agents for banks and large enterprises across Europe and the Middle East. If you want to hear how one performs on your own calls, you can test a voice agent with us.
