Blog

From simple, flow-based agents that help manage appointment booking and rescheduling, to more complex deployments that converse naturally with users, voice AI has applications across a wide variety of industries and use cases. The voice AI market is expected to exceed $22 billion in 2026, and Gartner forecasts that by 2029, agentic AI will autonomously resolve up to 80% of customer service issues without human intervention.

Adoption has been driven by improvements to model intelligence, as well as new capabilities unlocked by the adoption of speech-native models, rather than merely the urge to cut costs. At AI-native startups and established organizations alike, teams are building entirely new projects around voice agent capabilities, creating applications that were not previously practical nor scalable with human agents.

For developers building on or alongside these systems, the question is no longer whether voice AI is commercially viable — it's which architectural and capability decisions are worth prioritizing now.

Here is a grounded look at the trends shaping the space.

1. Emotional AI: Sentiment and Tone Detection

Most human speech carries emotional signals that a text transcript alone cannot capture — hesitation before giving a credit card number, frustration in a billing dispute, or urgency when a shipment has gone missing. Voice agents that respond to paralinguistic signals such as tone and cadence can better match (or balance) a user's heightened emotional state, delivering not just a better conversational experience, but a more positive resolution.

Traditional component pipeline architecture relies on transcribing a user's speech, allowing an LLM to reason on the resulting text. Thanks to modern ASR models, transcriptions are usually fairly accurate. But while transcription might perfectly capture the words spoken, it can't preserve the full richness of paralinguistic signals that heavily influence meaning.

For use cases that require a more natural conversational experience, teams are turning to speech-native voice AI systems. Unlike the component pipeline, speech-native models don't rely on transcription — instead, they perform reasoning directly on incoming audio. Among other benefits, this approach allows the model to reason on the full context of spoken audio, including non-transcribable paralinguistic signals. Voice agents are now being trained to detect those signals and respond to them appropriately, adapting to an individual user's apparent emotional state.

With speech-native models and modern training techniques, AI researchers are working to develop models with what the team at Sesame calls "voice presence," a qualitative description of spoken interactions that feel genuine and conversational.

2. Proactive Agents

Most voice agent interactions today follow the same basic pattern: a user initiates contact, the agent responds. That reactive model works well for inbound support or customer requests, but it leaves value on the table. In some use cases, the agent will have access to relevant information (via integrated systems) that the user hasn't yet thought to ask for.

For example, an avid shopper calling with questions regarding a return may not think to ask about a separate, more recent order that has just been flagged as delayed (and in fact, the customer may not yet be aware that their order is delayed). But a proactive agent with access to their order history can surface that information in the same interaction. Whether the brand in this scenario wishes to offer a discount or credit in apology or simply to inform the customer of the delay, the agent interaction will undoubtedly feel more personalized and more informative than the customer might've expected.

Proactive voice agents may also initiate contact or surface information based on event-driven triggers, such as a service outage, an annual check-up, or a shipment delay, without waiting for the user to call first. Production examples are already in deployment: agents that reach out when service degradation is detected in a user's area, or follow up with a patient after an appointment based on context from their recent visit.

Architecturally, this approach shifts the design pattern from a transactional request-response to a model that is primarily event-driven. The agent needs a way to connect to relevant data streams from scheduling systems, backend monitoring, or a CRM, as well as logic to evaluate when it's appropriate to initiate contact.

Rate limiting and user preference controls matter as much as the AI itself in these circumstances. An agent that reaches out daily in a well-intentioned attempt to book a dental cleaning is not likely to engender positive feelings on the part of the patient. Outreach that is too frequent or too aggressive is likely to erode the very trust that makes proactive contact valuable in the first place.

3. Real-Time Multilingual Translation

Language support has historically been treated as a localization problem: build an agent in one language, then commission translations for each additional market. That approach doesn't scale well, and it tends to produce uneven experiences — the primary language gets the most refinement, while languages added later often lag behind.

Modern voice agents are increasingly capable of detecting a caller's language mid-conversation and responding natively, including handling regional accents and mid-call language switches. Intent models are becoming language-agnostic, meaning a single trained workflow can serve speakers of different languages without duplicating business logic across separate deployments.

The core architectural question is whether to use a unified multilingual model or a language-detection and routing architecture. Unified models tend to win on latency and consistency; routing still has advantages where regional compliance requirements mandate separate data pipelines or where per-locale quality targets need to be tracked independently. In either case, for teams building for global audiences, designing for multilingual support from the start — rather than retrofitting it — tends to produce cleaner implementations.

4. Agentic Automation

Early voice agent deployments, while innovative, were largely transactional: the agent could answer questions, collect information, and route calls, but the actual work still happened downstream, handled by a human or a separate system. This was in part due to the novelty of the technology, but early voice agents relied on less-intelligent models than their more recent counterparts, so teams reasonably chose to restrict the tasks they'd allow a voice agent to do.

Agentic voice agents close that gap by executing multi-step tasks autonomously within the conversation itself; often processing a refund, updating an account record, booking an appointment, or triaging a support ticket without requiring a handoff.

This capability depends on the agent having well-defined access to external systems: APIs, databases, calendars, and other tools the agent can interact with in order to resolve a request. The orchestration layer needs to handle multi-step planning, failure recovery, and scope constraints — not just the happy path. Gartner projects that up to 40% of enterprise applications will embed task-specific agents by the end of 2026, up from less than 5% in 2025.

Audit trails and approval layers are not optional for anything touching sensitive operations. An agent with write access to a billing system can process a legitimate refund; it can also process an incorrect one. Documenting and designing the failure modes and oversight mechanisms before launch, rather than after the first production incident, is where most of the real engineering work lives.

See also: How 11x Outsourced Voice AI Innovation to Dominate their Market

5. Multimodal AI: Voice and Visual Channels

Voice is often the starting point of an interaction, but not always the right medium for its resolution. A caller asking about a contract clause, a billing breakdown, or a product comparison may be better served by a visual display than a verbal explanation. Increasingly, users expect the agent to make that transition from one medium to the next smoothly, rather than treating voice and screen as entirely separate experiences.

Leading platforms are beginning to treat voice as an orchestration layer that coordinates across telephony, messaging, and visual interfaces, rather than a standalone channel that loses context when a user switches surfaces. The practical challenge is context continuity: ensuring that session state, intent, and conversation history transfer cleanly when the interaction moves from audio to a screen-based interface.

Designing the cross-channel handoff before designing the voice experience may help produce cleaner implementations. Shared session tokens and unified state stores, rather than channel-specific ones, are the patterns most likely to hold up in production.

6. On-Device and Edge Architectures

Cloud-centric voice pipelines carry an inherent latency cost: audio travels to a remote model, inference runs, a response returns. For interactions that need to feel natural, that round trip is a meaningful constraint — and at scale, it's a financial one too.

The architectural response is a hybrid model: lightweight models handle acoustic perception and immediate intent classification on-device, while cloud inference takes over for complex reasoning and long-context tasks. In well-implemented hybrid systems, the majority of routine interactions can be resolved locally with near-zero latency, with the cloud reserved for requests that genuinely require it.

For teams targeting embedded or mobile deployments, the practical work involves designing the pipeline so that the boundary between local and cloud processing is explicit and deliberate — not an afterthought. Privacy is a secondary benefit that carries real weight in regulated industries: audio that never leaves the device is a meaningfully easier position to defend in a compliance conversation than one that relies on contractual data handling assurances from a third-party cloud provider.

Where This Leaves Developers

These trends are not isolated from one another. The most capable voice agents in production today combine emotional awareness, multilingual support, proactive and agentic behaviors, cross-channel continuity, and low latency within a single architecture. Building one component well is a reasonable starting point; building them to work together is where most of the challenging engineering work actually lives.

With the broader AI boom well underway, it's easy to forget that voice AI has only been a commercially viable solution for a few short years. But it helps explain why the gap between what is architecturally possible and what most teams have shipped remains significant.

The organizations and teams closing that gap between possibility and reality are not necessarily the ones with the largest models or the most unfettered access to GPUs. Instead, they are the ones that have made deliberate decisions at the integration layer, and designed for the failure cases before launch.

There’s been an explosion of Voice AI “agents” over the past couple of years, but the truth is that there is very little agentic about them. Most Voice AI agents that are deployed today are closer to classic IVR-style systems of the past than agentic systems of 2026 (though, admittedly, with much better TTS).

So, even though agentic use cases are well on their way to dominating the world of text models–think of Claude Code’s meteoric rise–many production voice-based systems remain stuck in late 2024.

There are two related reasons that explain this status quo.

Model intelligence gains often come at the cost of increased reasoning time

The most popular models used for production voice agents today include GPT-4o (released in May of 2024), GPT-4.1, and Gemini 2.5 Flash (both released in April 2025), all of which reflect training techniques from more than 18 months ago. A year and a half might not sound like much in real terms, but it’s several generations behind the current state-of-the-art. 

However, as models have gotten smarter, inference times have increased–for today’s frontier models, inference times can be in excess of several seconds. In a text-driven chat interaction, this less-than-instantaneous response time is unremarkable. But for voice agents, this increase in latency creates interactions that feel awkward, robotic, and stilted.

For voice agents powered by a component stack, ASR and TTS both contribute their own latency to the end-to-end reasoning pipeline. Older, legacy models perform considerably worse on reasoning, tool calling, and instruction following compared to the latest generation, but they offer one compelling advantage: faster reasoning. So by using an older model, teams stretch the overall latency budget further, albeit at the cost of model intelligence.

We lack a great harness for real-time interactions

The second problem is that we lack good harnesses for voice AI. In order to have a functional agentic system, you need a harness–a set of specialized primitives that wrap around the underlying model to handle details other than inference, such as memory usage or tool calling. 

Having sacrificed model intelligence in order to keep latency under control, most voice agents need some alternative means of ensuring desired behaviors. Less-intelligent models often struggle to cope with ambiguity, so many approaches rely on a set of deterministic rules (usually defined in a node builder or similar system) to govern the conversation. 

Deterministic guidance can help bridge the gap, improving instruction following behavior and generally keeping the model on track over the course of a conversation. But restricting the agent’s behavior to a narrow set of paths in this way often produces extremely unnatural conversational dynamics, and (ironically) can actually contribute to end-to-end latency. 

Compare this node-based approach with modern agentic harnesses, which assume ambiguity and design systems around how to elegantly handle that. What’s unique about the voice AI space is the demand for speed; agentic voice experiences don’t just need smart models and great harnesses–they need a system that works in real-time, sounds natural, and doesn’t have to wait on thinking tokens before responding.

The Foundations of Agentic Voice AI

So what do natural, agentic voice systems look like? They have three properties:

First, they’re fast. Speed is non-negotiable in agentic voice systems. If you’re not consistently under ~1s of end-to-end latency, you’re already too slow. If your agent is built using a component stack, your text LLM needs to consistently deliver a TTFT (time to first token) at or below ~500ms, to allow for the additional latency cost of ASR and TTS. Speech-to-speech systems, rather than component pipelines, are generally the best path to achieving the necessary speed. Ultravox, for example, is a speech-native system with an end-to-end latency of ~900ms. 

Secondly, they’re fluid. Agentic voice systems need to seamlessly call tools and manage the conversation state throughout a multi-turn interaction, without affecting speed or naturalness. Fluidity also means handling the ambiguity that arises in natural human communication–when the human speaker goes “off-script”, the agent needs to be able to adapt on the fly. This requires models that are exceptional at instruction following and tool calling, but also intelligent enough to respond gracefully to situations not explicitly described in the prompt. And realistically, if you’re not using 2026 models, you’re not going to get there.

And finally, agentic voice experiences need to be fluent. Users shouldn’t feel like they’re talking to a multi-faceted agentic system.  Behind the scenes, there may be multiple models, threads, and other complex patterns making sense of the conversation state, but conversing with the model should feel as natural as talking to another human.

At Ultravox, we’ve designed our system from the beginning around these principles–fast, fluid, fluent. We have the fastest, smartest model available today for speech, and we’re designing the most effective harness for managing complicated, long-running agentic voice conversations. Over the next few months, we’ll be releasing a series of articles on the design patterns, primitives, and system architecture that we believe will empower teams to design and build truly agentic voice AI systems.

Let’s take Voice AI into 2026.

Speech-to-speech is the future of voice AI–and as shown in the newly-released AIEWF eval, Ultravox’s speech-native model can outperform both frontier speech models and text models alike.

One of the ways we regularly test the intelligence of voice AI models is with standardized benchmarks, which test the model’s ability to perform tasks like transcribing audio, solve logic puzzles, or otherwise demonstrate reasoning based on spoken prompts. 

But modern voice agents need to do more than just understand speech–they need to reliably follow instructions, participate in extended multi-turn conversations, perform function calls, and reference information from a knowledge base or RAG in responses. 

Kwindla Kramer and the team at Daily have taken a new approach to creating a model evaluation framework that tests the capabilities that matter for production voice agents, beyond basic speech understanding. The AIEWF eval considers many of the practical requirements not tested in traditional speech understanding benchmarks–knowledge base use, tool calling, instruction following, and performance across a multi-turn conversation. 

Traditional speech understanding evals like Big Bench Audio help us understand how accurately a model understands speech. The new AIEWF eval measures how well a model can reason in speech mode and complete tasks that real-world voice agents need to do. 

Taken together, these evals demonstrate why speech-to-speech architectures are poised to overtake the component model for voice AI use cases.

Big Bench Audio: Evaluating speech understanding

Big Bench Audio is a widely-used evaluation framework that tests a model’s speech understanding, logical reasoning, and ability to process complex audio inputs accurately across multiple modes, including speech-to-speech and speech-to-text output. It’s a useful framework for quantifying a model’s inference capabilities given complex audio input. 

High scores indicate a model that effectively understands spoken language, which requires deep investment in architecture, training, and engineering (For the record, Ultravox v0.7 scores a 91.8 on Big Bench Audio–the highest score of any speech model available at time of writing.)

Naturally our team is excited about our achievement, but we think it’s worth being transparent about the limitations of the test itself. 

Big Bench Audio measures model capability in a narrow, relatively isolated context. Here’s an example question from the Big Bench Audio eval for speech-to-speech models (transcribed from original audio sample): 

Question: Osvaldo lies. Phoebe says Osvaldo lies. Candy says Phoebe tells the truth. Krista says Candy tells the truth. Delbert says Krista lies. Does Delbert tell the truth? Answer the question. 

A model that can correctly answer this question (Delbert does not tell the truth) demonstrates its ability to understand spoken audio as well as its ability to reason logically about speech, both of which are relevant if you’re building a voice agent. 

However, Big Bench Audio alone doesn’t tell you much about how well a model handles messy, real-time, multi-step workflows that voice agents typically face in the real world. There’s no tool calling, no complex instruction following, no background noise or interruptions. The eval is structured as a series of questions and answers that are evaluated independently, rather than as a multi-turn conversation.

In order to more accurately understand how well a speech model will perform in the real world, a test needs to measure how the model handles real-world demands. This is where things get interesting.

The AIEWF eval goes beyond speech understanding, and considers performance across: 

• Tool usage: Can the model correctly invoke external functions and APIs?

• Knowledge base integration: Can it retrieve and synthesize information from connected data sources?

• Multi-turn conversations: Can it maintain context and coherence across extended interactions?

• Task completion: Can it actually accomplish what users ask it to do?

These factors reflect many of the exact workflows that developers are building when they create voice agents for customer service, scheduling, information retrieval, and countless other applications. Instead of single-turn questions and answers, this framework tests reasoning across a set of 30-turn conversations, evaluating performance in individual conversations as well as the model’s consistency from one conversation to the next. 

When measured using the AIEWF multi-turn eval, Ultravox dominated the rankings for speech-to-speech models–in fact, it wasn’t even close. 

While a few frontier language models were able to outperform Ultravox on these practical benchmarks, it was with a pretty significant caveat: those models are far too slow for real-time voice applications. When it comes to voice agents, low latency is table stakes, not just a nice-to-have; a 2-second pause while waiting for the model to “think” about its answer can undermine the entire conversational experience. 

In short, Ultravox delivered frontier-adjacent intelligence at speeds suitable for real-time voice, delivering an overall experience that many developers might not have realized was possible.

Why speech-to-speech beats component stacks

For years, the standard approach to voice AI was building a component stack: a speech-to-text model feeds a language model, which feeds a text-to-speech model. Each component is built and optimized separately before being assembled into a pipeline.

This approach works(ish) for deterministic use cases, but it has fundamental limitations:

• Latency accumulates: Every component in the pipeline adds processing time. By the time you've transcribed, reasoned, and synthesized, you've burned through your latency budget.

• Errors compound: Transcription mistakes become reasoning mistakes, reasoning mistakes become response mistakes. There's no graceful recovery.

• Context gets lost: The prosody, emotion, and nuance in speech disappear the moment you convert to text. The language model never sees them.

Ultravox’s speech-to-speech architecture sidesteps all of these issues, because incoming audio never goes through an initial transcription step;  instead, the model performs reasoning directly on the original speech signal. The result is lower latency, fewer errors, and responses that actually reflect what the user said—not just the words, but the meaning. 

The Bottom Line

Traditional benchmarks are a useful signal of a model's fundamental ability to understand speech. But if you're building voice agents, it’s important to bear in mind that traditional benchmarks only tell part of the story.

If you're building voice agents, you need a model that excels at both: strong foundational capabilities *and* practical task performance. Ultravox delivers on both fronts. Highest scores on Big Bench Audio. Top speech model on real-world voice agent tasks. Fast enough for production use.

If you're evaluating models for your next voice agent project, the choice is clear. Ultravox isn't just competitive—it's the best all-around solution available today.

Ready to build with Ultravox? Get started here or contact our team to discuss your voice agent project.

There’s no such thing as “one-size-fits-all” when it comes to voice agents. Accent, tone, and expressiveness can make or break the end user experience, but the “right” choice depends entirely on the product you’re building. That’s why we’re excited to announce that Ultravox Realtime now allows users to build natural, conversational voice agent experiences using pre-built and custom voices using Inworld TTS 1.5. 

Emotionally expressive voices are the perfect complement to Ultravox’s best-in-class conversational voice AI, which is why we’ve teamed up with our friends at Inworld. Agents can now be configured to use any of Inworld’s pre-built voices in 15 languages; existing Inworld users who have created custom voice clones can also opt to assign a cloned voice to an Ultravox agent.

Try Inworld voices on Ultravox

To get started, simply log in to your existing Ultravox account (or create a new account).  

You can browse a list of all available Inworld voices and listen to sample audio by navigating to the Voices screen, where you can filter the list of available voices by language, provider, or both.

Once you’ve settled on a voice you like, you can either add it to an existing agent or create a new agent and select your preferred voice in the Agents screen.

Earn up to $100 in Ultravox Credits

To celebrate the launch of Inworld TTS 1.5, we’re giving away up to $100 in promotional credits for all our customers who run production voice agents with Inworld voices. 

How to qualify

1. Use Inworld voices for at least 60 call minutes during the promotion period: January 21 - February 28, 2026

2. Submit the usage survey form before March 13, 2026

3. Retain (do not delete) any eligible call records before April 1, 2026

4. For every 60 minutes of call time using Inworld voices during the promotional period, you’ll earn $1.00 in Ultravox account credits – up to $100!

All Ultravox customers (new and existing) on any paid or pay-as-you-go plan are eligible to participate, and there’s no cap on the number of customers who can take part in this promotion. 

For more information and FAQ, please visit this page. 

In our mission to deliver the best all-around conversational experience, Ultravox trained the world’s most advanced speech understanding model, capable of understanding real-world speech with no text transcription required. And Ultravox outperforms other speech-to-speech models in benchmarks that account for real-world requirements: accurate tool calling, reliable instruction following, and consistent low latency. If you haven’t given Ultravox a try yet, you can create your account here. 

Ultravox Realtime is now available as a speech-to-speech service in Pipecat. Use the deployment stack you’re used to with a model that accepts no compromise.

If you've built voice agents with Pipecat previously, you've faced a fundamental trade-off.

Speech-to-speech models like GPT Realtime and Gemini Live process audio directly, preserving tone and nuance while delivering fast responses. But when your agent needs to follow complex instructions, call tools reliably, or work with a knowledge base, these models often fall short. You get speed and native audio understanding, but at the cost of reliability.

Cascaded pipelines chain together best-in-class STT, LLM, and TTS services to get the full reasoning power of models like Claude Sonnet or GPT-5. But every hop adds latency, and the transcription step loses the richness of spoken language. You get better model intelligence, but sacrifice speed and naturalness.

Ultravox changes the equation

Like other speech-to-speech models, the Ultravox model is trained to understand audio natively, meaning incoming signal doesn’t have to be transcribed to text for inference. But unlike other models, Ultravox can match or exceed the intelligence of cascaded pipelines, meaning you no longer need to choose between conversational experience and model intelligence.

You don’t need to take our word for it–in an independent benchmark built by the Pipecat team, Ultravox v0.7 outperformed every other speech-to-speech model tested:

The benchmark reflects real-world conditions and needs, evaluating model performance in multi-turn conversations and considering tool use, instruction following, and knowledge retrieval. These results placed Ultravox ahead of GPT Realtime, Gemini Live, Nova Sonic, and Grok Realtime in head-to-head comparisons using identical test scenarios. Ultravox’s accuracy is on par with traditional text-only models like GPT-5 and Claude Sonnet 4.5, despite returning audio faster than those text models can produce text responses (which, for voice-based use cases, would still require a TTS step to produce audio output).

What this means for your Pipecat application

If you're using a speech-to-speech model today, switching to Ultravox will give you significantly better accuracy on complex tasks (tool calls that actually work, instructions that stick across turns, knowledge retrieval you can rely on) without giving up the low latency and native speech understanding you need.

If you're using a cascaded pipeline, you can switch to Ultravox and unlock the benefits of direct speech processing (faster responses, no lossy transcription, preserved vocal nuance) without sacrificing intelligence.

In either case, our new integration is designed to slot into your existing Pipecat application with minimal friction. 

• For users with existing speech-to-speech pipelines, the new Ultravox integration should work as a drop-in replacement. 

• For applications currently built using cascaded pipelines, you’ll replace your current STT, LLM, and TTS services with a single Ultravox service that handles the complete speech-to-speech flow.

Get started today

If you're already running voice agents in production, this is the upgrade path you've been waiting for. If you're just getting started with voice AI, there's never been a better time to build.

Check out this example to see how Ultravox works in Pipecat, then visit https://app.ultravox.ai to create an account and get your API key–no credit card required.

As 2025 winds down, we wanted to say, "thank you." Thank you for building with Ultravox, for pushing us to be better, and for being part of what has been, frankly, a mind-blowing year.

2025 by the Numbers

When we look back at where we started at the beginning of the year, the growth is hard to believe:

• 18x → How much larger our customer base is now than at the end of last year

• 38x → How much larger our busiest day was in 2025 vs. 2024

• 235 → The number of best-ever days on the platform in 2025

None of this happens without you. Every integration you've built, every voice agent you've deployed, every piece of feedback you've shared - it all adds up. You're not just using Ultravox, you're shaping it.

What's New

Before the holiday break, we wanted to get a few things into the hands of our users:

React Native SDK

Many of you asked for it, and it's here. Build native mobile voice experiences with Ultravox:

npm install ultravox-react-native

Source code and example app available in the repo.

Client SDK Improvement

Based on your feedback, we've made a change to how calls start: calls now begin immediately after you join them. No more waiting for end-user mic permission.

Why this matters: Inactivity messages now work as you'd expect. If an end user never starts speaking, the inactivity timeout can end the call automatically. This keeps those errant calls to a minimum and saves you money.

Call Transfers with SIP

Your agents can now transfer calls to human operators with built-in support:

• coldTransfer — Immediately hand the call to a human operator. No context, instant handoff. Docs →

• warmTransfer — Hand off to a human operator with context about the call, so they're prepped and not going in cold. Docs →

Full guide: SIP Call Transfers →

Here's to What You'll Build Next & A BIG 2026

Voice AI is hitting an inflection point. The technology is finally good enough (fast enough, natural enough, affordable enough) to power experiences that weren't possible even a year ago.

Our 0.7 model release has been benchmarked against every major competitor (Gemini, OpenAI, Amazon Nova) and Ultravox leads in both speed and quality. When you factor in our $0.05/minute pricing, the choice becomes pretty obvious.

Thank you for being a customer. Thank you for believing in what we're building. And thank you for an unforgettable year. We have been hard at work on some new capabilities we can’t wait to get in your hands! Stay tuned for lots more to come in 2026!

Happy holidays from all of us at Ultravox.

Today we’re excited to release the newest version of the Ultravox speech model, Ultravox v0.7. Trained for fast-paced, real-world conversation, this is the smartest, most capable speech model that we’ve ever built. It’s the leading speech model available today, capable of understanding real-world speech (background noise and all!) without the need for a separate transcription process.

Since releasing and sharing the first version of the Ultravox model over one year ago, thousands of businesses from around the world have built and scaled real-time voice AI agents on top of Ultravox. Whether it’s for customer service, lead qualification, or just long-form conversation, we heard the same thing from everyone: you loved the speed and conversational experience of Ultravox, but you needed better instruction following and more reliable tool calling.

We’re proud to share that Ultravox v0.7 delivers on both of those without sacrificing the speed, performance, and conversational experience that users love (in fact, we improved inference performance by about 20% when running on our dedicated infrastructure). 

One of the most important changes in v0.7 is our move to a new LLM backbone. We ran a comprehensive evaluation of the best open-weight models and ended up choosing GLM 4.6 to serve as our new backbone. Composed of 355B parameters and 160 experts per layer, it substantially outperforms Llama 3.3 70B across instruction following and tool calling. As always, the model weights are available on HuggingFace. 

We’re also keeping the cost of using v0.7 on Ultravox Realtime (our platform for building and scaling voice AI agents) the same at $.05/min. That’s state-of-the-art model performance with speech understanding and speech generation (including ElevenLabs and Cartesia voices) for a price that is half the cost of most providers. 

You can get started with v0.7 today either through the API or through the web interface.

Frontier Speech Understanding

Ultravox v0.7 is state-of-the-art on Big Bench Audio, scoring 91.8% without reasoning and an industry-leading 97% with thinking enabled.

On VoiceBench, v0.7 (without reasoning) ranks first among all speech models. When reasoning is enabled, it extends its lead even further, outperforming both end-to-end models and ASR+LLM component stacks.

When measured for speed, the Ultravox model running on our dedicated infrastructure stack for Ultravox Realtime achieves speech-to-speech performance that is on-par or better than equivalent systems:

Numbers taken from Artificial Analysis' Speech-to-Speech "Speech reasoning vs. Speed" [source]

Ultravox Realtime Platform Changes

Ultravox v0.7 will become the new default model on Ultravox Realtime starting December 22nd, but you can start using it today by selecting ultravox-v0.7 in the web UI or setting the model param explicitly when creating a call via the API. 

This default model update will affect all agents using fixie-ai/ultravox as well as agents with unspecified model strings. If you plan to migrate to this model version, we recommend starting testing as soon as possible, as you may need to adjust your prompts to get the best possible experience with your agent after the model transition.

While we think Ultravox v0.7 is the best-in-class option, we’ll continue to make alternate models available for users who need them. So if you’re happy with your agent’s performance using Ultravox v0.6 (powered by Llama 3.3 70B) and don’t plan to change, or if you just need more time for testing before migrating to Ultravox v0.7, you’ll be able to decide when (or if) your agent starts running on the latest model. 

You can continue using the Llama version of Ultravox (ultravox-v0.6), but you’ll need to manually set the model string to continue using it. If your agent is already configured to use fixie-ai/ultravox-llama3.3-70b or ultravox-v0.6, then no changes are required–your agent will continue to use the legacy model after December 22.

We’ll also be deprecating support for Qwen3 due to a combination of low usage and sub-optimal performance (GLM 4.6 outperforms Qwen3 across all our tested benchmarks). Users will need to migrate calls and/or agents to another model by December 21  in order to avoid an interruption in service. Gemma3 will continue to operate as it does currently.

Q: I followed the instructions and cloned my voice to use with my AI agent, but my voice clone doesn’t really sound like me–why isn’t it working?

We’ve encountered this type of user question often enough that we thought it was worthwhile to do a deep dive into how voice cloning works. 

The short answer is that there are lots of reasons why your voice clone might sound noticeably different from your actual voice–some of these are easy to fix, while others are due to more structural nuances of how voice cloning and text-to-speech models work.

The basics of voice cloning

Humans learn language through a complex process, first by hearing and recognizing word sounds, then linking words to real-world concepts, and eventually grasping more abstract concepts like the pluperfect tense. So a four-year-old might stumble while pronouncing “apple” or “toothbrush” but will generally recognize that those words correspond to physical objects in the world around them.

Training a text-to-speech (TTS) model is wildly different. Although modern models increasingly incorporate semantic information, their understanding of individual words or sentences remains limited, unlike human comprehension. Instead, models learn the mappings from text sequences to audio waveforms via the patterns of some intermediate representation, such as spectrograms or quantized speech tokens. From there, the model can reproduce patterns that sound like human speech. A TTS model can accurately pronounce the word “toothbrush”, but without any human-like understanding of the object that word represents.

An example of a derived spectrogram, waveform, and pitch contour based on an adult male speaking voice.

Training a TTS model relies on aligned text and audio pairs: give the model a text snippet, and a corresponding audio clip of a human reading the text, and then repeat the process a few million times. Eventually, the model output for a given text sample will sound convincingly like a human reading the text aloud. 

The quality and fidelity of speech generation in these models varies between languages because there is wide variety in the availability of text-audio samples for different languages and dialects. Training data for languages like English, Spanish, and Mandarin is extensive and widely available, while Farsi, Yoruba, and Khmer are considered low-resourced languages, meaning they have little in the way of paired text-audio data to train on.

Many indigenous and endangered languages have no standardized TTS datasets at all. If you speak one or more low-resourced languages, please consider contributing to Mozilla’s Common Voice project by recording, verifying, or transcribing audio samples.

Once the model has been trained on a sufficiently large and diverse dataset, it can synthesize the voice of a new human speaker from a remarkably brief sample–typically 10 to 60 seconds of audio, depending on the model’s training–without any additional fine-tuning or adaptation. This is accomplished through a process called zero-shot voice cloning.

In zero-shot voice cloning, the model analyzes a short reference clip to capture a speaker’s vocal traits–qualities like timbre, pitch, and nasality–and encodes these traits as explicit acoustic features or latent embeddings. This encoding can then guide speech generation, so the model can produce new utterances in the same voice from text alone. Some modern TTS models use the reference audio sample as a prompt prefix, but there’s no speaker-specific fine tuning or training involved.  

The advantage of the zero-shot method is that it’s easily accessible–almost anyone can record a 60-second audio clip, and because there’s no fine-tuning of the generated speech beyond the initial embedding, it’s quick to do. The cloned voice won’t be a perfect match (due to the absence of fine tuning as well as for some reasons we’ll discuss below), but if what you need is a reasonably good replica of a speaker’s voice, the zero-shot method is a great option.

Limitations of the zero-shot voice cloning method

There are some aspects of human speech that even the best TTS models will struggle to replicate using the zero-shot method.

Prosody

Prosody refers to the melody and rhythm of speech; patterns like pitch, timing, and emphasis that make a speaker sound natural and expressive. While a model can often capture the basic qualities of a person’s voice, such as pitch and timbre, it’s much harder to consistently reproduce their prosody. Most speakers will share some general intonation patterns, like raising pitch to indicate a question, but individual speakers will also develop unique habits of pacing, stress, and tone that vary depending on mood and context. Depending on how the text prompt is written (for example, using punctuation marks to denote pauses or capitalization to mark emphasis) generated speech might sound like the right voice, but with the wrong delivery.

Local and complex accents

Even for languages with extensive training data available, specific local or regional accents might be under-represented. For example, French and English are both among the most high-resource languages for TTS, but Belgian French or Scottish English might appear far less frequently in training data. The challenge is even greater for regions with extensive bilingualism (like the largely German-Italian bilingual population of the South Tyrol region), as speakers will naturally mix features like phonetic and rhythmic patterns from multiple languages.

Expressiveness and emotion

For obvious reasons, a 60-second audio sample will generally not capture the full range of a speaker’s emotions (excited, sad, sarcastic, defeated) for the model to reference. While most models can reproduce emotional styles that they’ve been trained on, their ability to model a speaker’s expressiveness is limited if emotion and speaker identity are not clearly disentangled. In some cases, the cloned voice might inherit the emotional tone present in the reference sample, or simply default to an emotionally neutral delivery.

There’s one additional factor that might explain why your voice clone sounds “off”, although it’s not unique to voice cloning. When you speak, you usually hear your voice through two pathways: sound waves traveling through the air into your ears (air conduction), and vibrations traveling directly through your skull to your inner ear (bone conduction). 

Solid materials like bone conduct lower frequencies more efficiently than air, so bone conduction essentially gives you an internal bass boost, making your voice sound richer and deeper when you hear yourself speak.

Audio recording, however, captures only the air-conducted sound. When you play it back, you’re missing that extra resonance from bone conduction, so your voice might sound “thinner” and higher-pitched than what you’re used to. Your brain is likely used to the blended version, but audio recordings only give you the external version that everyone else hears. 

Since a voice clone embedding is also generated using only air-conducted sound from an audio sample, it might be a pretty close match to the way other people hear your speaking voice, even if it sounds a bit odd compared to the way you’re accustomed to hearing yourself.

Tips on making a better voice clone

While generated speech might not be a perfect replica of your voice, there are some steps you can take when producing an audio sample that will improve the overall quality of your voice clone: 

• Make sure your recording is free of background noise, especially sound from other voices, music, etc. You should be the only speaker audible on the sample. 

• The acoustics of the room you record in also matter–if you make your recording in a large, relatively empty space, artifacts from the background reverberation in the room can be inherited by your voice clone. 

• Maintain a natural speaking pace and tone throughout the entire sample recording–it might be helpful to create a brief script in advance and read it out loud.

• The quality of your audio input device matters–recoring on your default laptop hardware often produces a much lower-quality sample than recording on a high-quality external microphone.

  
  

We recently announced that we're open sourcing UltraVAD, the smart endpointing model that we use for running production traffic in Ultravox Realtime. UltraVAD is a multimodal model, meaning it uses a combination of transcripts and audio to evaluate whether or not a user has finished speaking.

We started work on UltraVAD by designing around two key principles:

• Audio-native: the model should process direct audio input without reliance on a transcription step

• Context-aware: the model needed to reference dialog history to better understand contextual meaning

  
  

In this article, we’ll be taking a deeper dive into how the team built UltraVAD, and what we learned from some of our earlier versions.

Lessons from v1: a supervised binary classifier

Our first version used a supervised learning approach. We trained an LLM to work as a binary classifier and fine-tuned on conversational samples with binary labels. Each training sample consists of a previous turn, and last user turn, along with an end of turn true/false label. We used a foundation model like gpt-4o to classify these samples.

*Previous turn:* “How’s the weather looking?”
*Current turn:* “It was warm today.” → **True

*Previous turn:* “How’s the weather looking?”
*Current turn:* “It was, um—” → **False

Unsurprisingly, we found the v1 approach struggled in a few key ways:

• Context awareness. Numbers, lists, and partial utterances look like stops in isolation: “My number is 408…” vs “My area code is 408.” It’s hard to gather context-dependent samples without missing edge cases.

• Dependence on an outside classifier. Our v1 leaned on a foundation model to classify samples, but we found that these foundation models generally don’t have good endpoint distributions for languages other than English.

• Ambiguous boundaries. Assigning binary labels in the samples forces boundaries on turns that are ambiguous, which adds noise to the training data.

We ultimately realized that endpointing is difficult to capture with binary-labeled samples; it’s better modeled as an emergent distribution learned over a conversational corpus.

Improvements in v2: self-supervised next-token prediction task

In our second iteration, we tried a new approach: teaching an LLM to perform self-supervised training with an end-of-turn (EOT) token, then making this process audio-native with an Ultravox projector. The idea was to allow the model to use both textual context and audio, which would outperform models that used text or audio alone and make it easier to scale to new languages. 

For this iteration, we started with an existing Llama-8b-Instruct model. To refine its training, we synthetically generated multi-turn conversational data with additional end-of-turn <eot> tokens placed after valid stopping points in the conversation. 

This process taught the LLM a distribution over the end-of-turn token, and we could then use that distribution along with a decision boundary for end-of-turn classification.

<assistant> "How's the weather looking?" <user> "It was, um- warm today" <eot

With this approach we could apply self-supervised finetuning¹ on an unbounded corpus of conversational data, instead of supervised learning on carefully curated true/false pairs.

We also found that this approach extends well to new languages. In the binary classification approach, translating the sample could actually change the boundary, so we would have to rerun the end-of-turn classification on new language datasets. 

In addition, for some languages (such as Spanish), foundation models have poor end-of-turn classification abilities, meaning our v1 approach couldn’t support those languages. With this new v2 approach, any conversational corpus could be trained on, which meant we don’t have to rely on an external classification model to distill from. Translating the training corpus preserves the endpointing distributions across languages more faithfully than reclassifying with a foundation model.

For each additional language we wanted to support, we simply applied a translation transformation to our existing corpus and trained on it. Now, our endpointing model supports 26 languages, and can be easily extended to support more in the future.

Adding the Ultravox projector to make our model audio native

After text-based finetuning, we integrated the Ultravox audio projector—imbuing our endpointing model with audio-native abilities.

Our training approach:

1. Initialize the projector using a pretrained Ultravox projector, already optimized for noisy, real-world speech.

2. Finetune on synthetic dialogue data, aligning the audio embeddings with our end-of-turn transformer.

Because the projector is pre-trained for robustness (handling background noise, varying mic quality, and overlapping speech), the endpointing model inherits these strengths, and delivers reliable turn-taking even in the same chaotic conditions that Ultravox is accustomed to.

Deploying UltraVAD in production

Now that we have the endpointing model, how do we actually use it in a realtime voice service? A naive way would be to run the endpointing model after every audio packet that streams in. Not only is this expensive–we’d be running inference every 30 milliseconds–but we’d also take an additional latency hit if we only run the LLM+TTS after getting a response from the endpointing model.

Instead, we use VAD as the first line of defense in our Ultravox Realtime platform. VAD models (e.g. Silero VAD) look at audio stream packets and give each chunk a “likelihood of human speech” score. After a specified threshold of “no human speech” packets is reached, we speculatively run a forward pass on UltraVAD with the last user audio turn, along with the textual conversation history.

In order to keep latency low, we also speculatively run our Ultravox model at the same time. Generally we receive a reply from UltraVAD before the first packets from our TTS model have come back (latency here is around 500ms). Therefore, as long as UltraVAD’s forward pass stays under the time-to-first-audio packet, we don’t “pay” for that latency. Given this latency headroom, we chose to build a larger, more powerful model rather than a smaller one that sacrifices accuracy for speed.

What's Next?

We’ve got more improvements for UltraVAD in the works, including improvements to how the model handles paralinguistic cues in conversation as well as group conversations. 

In the meantime, model weights for the current version of UltraVAD are open source and available on Hugging Face.

If you'd rather skip straight to the fun part and try it out yourself, you can also check out a live demo of the Ultravox assistant, or create a free Ultravox account to and follow our Quickstart guide to start chatting with your own custom voice AI agent!

—

¹ We apply a cross-entropy loss on all tokens, as opposed to just the user response.

² Threshold: 0.1 for UltraVAD and 0.5 for SmartTurnV2

UltraVAD, our version of a smart endpointing model that we use for running production to Ultravox Realtime, is now open source!

UltraVAD is audio native, meaning it doesn’t rely on automated speech recognition (ASR) and it leverages conversational context to make highly accurate predictions about when a user is done speaking. In our analysis, it's the most accurate VAD model for situations that require conversational context to make an accurate prediction.

For reference, we’ve shared our model weights on Hugging Face.

The turn-taking problem in Voice AI

At its core, turn-taking is a question of knowing when to speak (or when not to speak) in a conversation, and it's one of the harder problems in Voice AI. Most humans do this effortlessly, leaning on a blend of cultural and conversational context, linguistic signals, and paralinguistic cues like pauses, intonation, and pitch. But what feels instinctive to people becomes three engineering problems¹:

1. Endpointing - detecting whether a user has finished speaking

2. Interruption handling - detecting whether a user is attempting to interrupt the speaker or backchannel (verbal cues like “uh-huh” or “yeah” that indicate a listener’s engagement)

3. Multi-party turn-taking - handling multiple speaker channels

In this blog, we’ll mainly focus on the first of these: endpoint detection.

In the past year most voice AI systems have graduated from simple silence-duration heuristics to neural endpointing models.

At Ultravox.ai, we frame endpointing as a next-token prediction task. By leveraging our existing Ultravox audio projector, we fuse audio, dialog context, and LLM semantics to estimate end-of-turn probability in real time—reliably deciding when a user is done speaking across 26 languages.

Existing approaches to endpointing

The simplest approach to endpointing streams audio packets through a Voice Activity Detector (VAD) (e.g. Silero VAD) which assigns a “likelihood of human speech” score. After a threshold of “no human speech” packets is reached, the orchestration layer deems the speaker’s turn over and triggers the LLM to respond.

The problem with this approach is that duration of silence is the only signal, which leads to a higher rate of premature cutoffs. Real conversations rely on richer semantic and contextual cues.

Text-native neural endpointing systems operate on transcripts, waiting for ASR output in order to decide if the user has finished speaking.  Because it relies on transcripts, this method sacrifices paralinguistic awareness–cues like changes in pitch, intonation, or rhythm. Minor mis-transcriptions (think of homonyms like “break” and “brake”) can completely change the meaning of a sentence and lead to inaccurate endpoint classification. And there’s an inherent bottleneck introduced by ASR transcription latency as well as the overall reliability of your ASR vendor chain. The result can feel unnatural at scale: long pauses, awkward interruptions, broken phrases.

Some systems (e.g., Krisp, Pipecat) use audio-native neural endpointing, skipping ASR and processing raw audio directly. This audio-native approach retains the kind of paralinguistic cues that text-native neural endpointing misses. However, these models only consider the most recent user turn, meaning they tend to struggle when contextual meaning determines whether a speaker’s turn is complete:

In the above example, prior dialog provides essential context in the conversation–without it, even a human listener would struggle to know for sure whether the speaker is likely to continue.

Multimodal models combine transcripts and audio to leverage both semantic context and audio signals. Audio-native processing allows for direct audio input without a transcription step, while preserving the dialog history allows for better context awareness. After evaluating other options, we chose to move forward with this model type for UltraVAD.

Evaluating performance

At the time of this writing, there are not many ways to evaluate neural endpointing models. Pipecat’s Smart-Turn V2 offers helpful single-turn datasets, but we saw a gap for multi-turn, context-dependent evaluation. So we’re releasing our own benchmark focused on contextual turn-taking.

For our benchmark, we synthetically generate these context-dependent samples, and ask a foundation model to label the last user turn of these samples. Then we hand-check these labels to make sure they are correct.

On a held-out set of 400 context-dependent samples, we compared UltraVAD to Smart-Turn V2 (audio-native). We chose Smart-Turn V2 because it is the only other open source audio-native endpointing model. The default recommended thresholds³ are set for both models, and here are the results.

For consistency, we also evaluate UltraVAD on Smart-Turn v2’s single turn datasets:

Comparing their reported score to our aggregate eval score on their test set, the results are within a 1 percentile difference, while there is nearly a 20 percent improvement in the area under the ROC curve (AUC) score for the context-dependent samples.

Open Sourcing UltraVAD

After building and testing several iterations of a multimodal model, we landed on a version that we felt confident including in our Ultravox Realtime platform. (For more details on that development process, see Context-aware, audio-native endpointing: How we built UltraVAD). If you’re using the Ultravox.ai realtime stack, this model is already turned on by default. 

We are also pleased to announce that an open source version of UltraVAD’s weights is now available on  Hugging Face! 

Although the current model is audio native, we still need to improve its ability to use paralinguistic cues in conversation. Work is currently underway to train our projector to recognize intonation, pitch, and pauses, which will improve performance on semantically ambiguous cases. With that in mind, we will be continuously pushing updates to make this model better. 

We also plan on improving our turn-taking model in the future to handle interruptions, backchannels, and group conversations - so stay tuned!

—

¹ We chose to target these three dynamics as they are the most tractable and impactful turntaking problems right now, but there may be other more nuanced parts not covered here. Ie. exponential backoff when two parties both decide to talk at the same time.

The months leading up to GPT-5's release witnessed an dizzying wave of new language models. This summer alone introduced Kimi-K2, Grok 4, GLM-4.5, Qwen Instruct, GPT-OSS, GPT-5 — and we're only halfway through the year.

Each technical report heralds another state-of-the-art model, but this rapid pace raises an critical question: how can we effectively evaluate these models for real-world applications?

Ultravox is an open-source speech language model built on existing open-weight language models. Unlike traditional systems that rely on a separate transcription step, Ultravox is trained to understand speech directly. This design makes it easy to integrate the latest LLM breakthroughs into our platform.

Though the pace of new releases excites both us and our customers, we’ve discovered our benchmark-based excitement diminishes once we evaluate them in real-world conditions. To help inform our decision about which of these models are best to offer to our customers, we needed a more systematic way to evaluate models on real-world performance.

To accomplish this, we built a benchmark tailored to voice agents (though we suspect it’s useful for conversational agents more generally). Our goal is to measure how well a model can engage in rapid, natural dialogue while still following instructions, calling tools, and minimizing hallucinations. In this blog post, we'll share why we built VoiceAgentBench, what it measures, and how we interpret the results (if you just want to see the numbers, click here).

Which Benchmarks Should I Trust?

When Claude 3.7 Sonnet was released in February, many were underwhelmed by its seemingly modest benchmark improvements. Their blog post stated:

…in developing our reasoning models, we've optimized somewhat less for math and computer science competition problems, and instead shifted focus towards real-world tasks that better reflect how businesses actually use LLMs

The simultaneous release and subsequent popularity of Claude Code revealed the truth—benchmarks don't tell the whole story. While new benchmarks provide fresh perspectives on model capabilities, they also create targets for future models to aim for. Hitting these targets generates buzz, but real-world applications extend far beyond these narrow measurement points across a much broader landscape.

Beyond concerns like data contamination, this framing—along with insights from experts in post-training(1)—suggests that benchmark performance is increasingly a choice made during training. As a result, their diagnostic value diminishes over time, enabling a steady stream of state-of-the-art results that incrementally push performance forward.

So what can we do to counter leaderboard-chasing and find meaningful signals? We see two effective approaches:

1. Compare performance across multiple benchmarks (we like Artificial Analysis for this)

2. Develop a custom benchmark for your own application

While we carefully monitor various benchmarks, we've discovered that none perfectly addresses our users' specific needs. A custom benchmark remains the most effective way to measure the capabilities that truly matter for voice agents.

Textual Evaluations for Voice Agents

Ultravox works by pairing a speech encoder and multimodal adapter with pretrained language models. We train our adapter using a teacher-student paradigm, such that the language model treats speech input in the same way as text, transferring all of the powers of unimodal LLMs into the audio domain. To choose language models for Ultravox integration, text-domain performance gives us a sneak peek on how a trained Ultravox model will perform in the audio-domain.

For most agentic AIs, three capabilities matter most: tool calling, instruction-following, and producing hallucination-free responses. Benchmarks exist for each but reported scores are often treated as absolute, when reality they vary greatly with context. A model might call the right tool every time in single-turn experiments, but fail when a tool call is required in the middle of a long conversation.

Voice agents face especially demanding conditions:

1. Multi-turn Conversations: Guiding dialogue toward goals across several turns, while tracking state and anticipating user behavior.

2. Long-Horizon Tool Calling: Decisions about which tools to call (and when) often span multiple turns, not just one.

3. Diverse, Real-World Scenarios: Handling varied tools, prompts, tasks, languages and user behaviors—including rare cases—without hallucinating.

If we base our evaluation on existing benchmarks, we want to make sure that they are measuring agentic capabilities under these conditions.

Do Existing Benchmarks Match These Conditions?

Of the many benchmarks that exist for agentic AI, two of the most widely reported are:

• TauBench: Tests multi-turn, long-horizon assistance in only three manually constructed agents, each with multiple simulated users. Users drive conversation, and tasks are intentionally complex, so even the top models score below 50% on the reliability focused pass^4 metric—a reflection of the difficulty, not necessarily poor everyday performance.

  • Main limitation: the lack of agent diversity. With only three, TauBench can’t reflect the breadth of situations a voice agent will face. It’s excellent for stress-testing but tells us little about performance in the wide variety of scenarios where a voice agent should succeed.
    
    

• BFCLv3: Evaluates multi-turn function calling with crowd-sourced tools in multiple languages. To control variability, it uses fixed “ground-truth” trajectories (rather than simulating users) and resets conversation history to that state before each model turn—reducing long-horizon challenges into a series of single-turn calls.

  • Main limitation: while it covers more agent diversity, resetting to the ground-truth state strips the benchmark of the reasoning nuance and complexity required for long-horizon tool calling.

These are far from the only benchmarks in the field—but because they’re so frequently reported, they shape much of the public narrative about model capability. Both constrain their setups to make results verifiable. In doing so, TauBench sacrifices agent diversity, and BFCLv3 sacrifices long-horizon realism. Neither captures tool use, hallucination resistance, and instruction-following in the unpredictable, multi-turn reality of voice agents.

That’s why we built our own evaluation framework and an internal benchmark—to test these capabilities under the same varied, real-world conditions our agents will actually face.

VoiceAgentBench: Scalable and Diverse Evaluations

To go beyond leaderboard chasing, we built VoiceAgentBench—a benchmark designed specifically to test models in the conditions that voice agents actually face: multi-turn conversations, long-horizon tool use, and diverse real-world scenarios.

Framework Design

Existing benchmarks rely on hand-labeled ground-truth calls, which limits scalability and diversity. We designed the simulation framework for VoiceAgentBench to overcome this:

• AgentProfiles and UserProfiles: We simulate both sides of the conversation using LLMs, where an AgentProfile defines the agent prompt and tools, and a UserProfile defines the user prompt. Together they form a scenario.
  
  

• ToolResponder: Instead of static databases, we simulate realistic tool outputs via structured JSON responses. This makes the framework scalable to arbitrary tools without hand-curation.
  
  

• LLM-as-Judge: Following recent work [cite], we use LLMs to grade conversations. We craft three rubrics: tool calling, hallucination, and instruction-following. Each rubric breaks down into criteria that pass/fail, giving us fine-grained insights. Overall score on a single scenario represents the average pass rate among these three rubrics.

This setup removes the biggest bottlenecks of manual annotation and enables us to test a much wider range of agents than TauBench or BFCLv3. This allows us to create AgentProfiles and UserProfiles for evaluation that match the full diversity that we find in our call logs, and continue to update those as voice agent use-cases evolve.

Figure 1: The VoiceAgentBench framework design, consisting of scenario sampling (left), trajectory rollouts (center), and evaluations by LLM-as-judge models (right).

Results

From select partners in our logs, we sampled 36 recent AgentProfiles and paired each with two UserProfiles, yielding 72 scenarios. We fixed these scenarios as our VoiceAgentBench benchmark and scored multiple leading proprietary and open-source models on the scenarios.

We report the distribution of overall scores for leading reasoning and non-reasoning models in Figure 2, where we show surprising mediocrity from a few of the most talked-about open source models (Kimi K2, Qwen 3 235B Instruct, GPT-OSS-120B). These models do not demonstrate significant advances over our existing Ultravox lineup. Conversely, GLM-4.5 stands out as a competitive open-source alternative to proprietary models on real-world VoiceAgentBench scenarios.

Figure 2: Box plots showing the distribution of overall scores for each non-reasoning (left) and reasoning (right) model on VoiceAgentBench. The labeled numbers represent the median overall score for each model on the dataset, and the boxes represent the interquartile range (IQR), with whiskers to 1.5x the IQR and outliers circled.

These surprising results made us wonder—are we missing something? While the overall trends on VoiceAgentBench aligned with our intuition, some open-source models were not meeting our expectations. We compared mean mean scores for each model in VoiceAgentBench to their TauBench Airline scores in Figure 3.

When comparing VoiceAgentBench scores against TauBench, we find strong correlation (r=0.83), validating that our benchmark reflects real capability. At the same time, it’s clear that narrow benchmarks are missing some important dynamics. Models that excel on TauBench do not always achieve corresponding success on VoiceAgentBench, and open-source models in general appear to underperform in real-world scenarios. This gap underscores the importance of having application-grounded evaluations—benchmarks tuned for comparability can miss how models behave in the wild.

Figure 3: Comparison between mean VoiceAgentBench scores and TauBench Airline accuracy (pass^1). Pearson’s r=0.83. (*) **indicates that reasoning levels are not controllable in the TauBench environment

From these results, we can draw some high-level takeaways:

• The best open weight models are catching up (but most seem optimized for leaderboard performance over real-world performance). GLM-4.5 nearly matches the top proprietary non-reasoning models (gpt4o, gpt-4.1), showing how quickly open models are improving. But the picture shifts across benchmarks (Fig 3): open-source models that shine on narrow leaderboards like TauBench may not always perform better on VoiceAgentBench’s real-world scenarios.
  
  

• GPT-OSS is not suitable for real-time voice. Like everyone, we were excited by OpenAI’s initial release of GPT-OSS line of models. Unfortunately, we did not find the high-reasoning variants to be a step-change in capabilities, and the low-reasoning variants (still not clearly suitable for low-latency interactions) performed even worse on VoiceAgentBench.
  
  

• Reasoning models excel—but are too slow. Latency is uniquely important in voice agent deployments, but model advancements have been more agnostic to this requirement. There is a need for additional investment and development to push non-reasoning models forward.
  
  

• Deployment matters. Our Llama 3.3-based Ultravox model consistently outperforms vanilla Llama. Part of this comes from integration improvements (chat template, tool parser), but also from sampling bias: the AgentProfiles in the dataset come from Ultravox logs, where prompts were tuned for Ultravox’s template. This sampling bias favors Ultravox, but it highlights a broader lesson: prompt optimization can meaningfully raise performance, meaning most other models’ scores should be seen as lower bounds.

While some open-source models underperformed on VoiceAgentBench, their relative strength on TauBench highlights a different story: these models are highly capable but often optimized for leaderboard metrics rather than real-world reliability. This reflects the incentives shaping development—open providers race to win adoption through benchmark scores, while proprietary teams may prioritize robustness and user experience to drive retention.

At Ultravox, we see an opportunity to close that gap. VoiceAgentBench brings evaluation closer to the reality of user interactions—messy, diverse, and long-horizon. With our simulation framework, we can also shift the focus of model training from synthetic benchmarks to practical outcomes. For us, the most desirable result isn’t topping a leaderboard—it’s enabling models that perform reliably in real-world voice applications.

Looking Forward

Open-source models are improving at a remarkable pace. Just a year ago they trailed far behind proprietary ones, but today, open models like GLM-4.5 are approaching parity with their closed counterparts. Yet VoiceAgentBench shows that leaderboard gains don’t always carry over to long, messy, real-world conversations—where reliability matters most.

For Ultravox, this creates both a challenge and an opportunity. The challenge is that headline benchmarks can obscure what actually matters in production. The opportunity is that open-source models, precisely because they are flexible and adaptable, can be pushed further. Our low-latency speech stack makes it possible to deliver these models directly into customer applications, and VoiceAgentBench gives us the visibility to understand where they succeed, where they falter, and how to close the gap.

We’re building VoiceAgentBench not just for ourselves but for the developer community. Our simulation framework design makes our approach scalable to diverse agents and scenarios. That’s why we’ll be rolling out the framework behind VoiceAgentBench inside the Ultravox platform. Paired with Blocky for automated prompt refinement and future post-training advances, this will create a new kind of voice-agent development loop: one where evaluation, optimization, and deployment reinforce each other.

Benchmarks will always make headlines. But for us (and for our users) the real test is performance in live conversations. VoiceAgentBench is how we cut through the noise, focus on what counts, and help developers build the next generation of voice agents.

The voice AI industry is being built on deprecated architecture.

Most voice AI platforms (from customer service bots to AI phone agents) are built on the same foundational architecture: Automatic Speech Recognition (ASR) converts speech to text, a Large Language Model (LLM) processes the text, and Text-to-Speech (TTS) converts the response back to audio.

This ASR→LLM→TTS pipeline seems logical. It's modular, leverages existing technologies, and appears to work. But it's fundamentally flawed, and the cracks are starting to show in accuracy, latency, scalability, conversational realness, and cost.

Component stacks limit your product velocity and performance:

• You’re bottlenecked by the weakest service (often ASR)

• You can’t adapt responses based on acoustic context

• You incur latency at every step

• Your cost and complexity grow linearly with traffic

Why? Because this architecture discards the very thing that makes speech special.

Speech is Not Just Words

When humans communicate, we don't transcribe speech into text in our heads before understanding it. We process the acoustic signal directly, using context clues that exist only in the original audio: tone, emphasis, hesitation, emotional state, and the subtle rhythms that make conversation feel natural.

The moment you convert speech to text, this critical information vanishes forever. No amount of sophisticated prompting or fine-tuning can recover what was lost in that first conversion step.

Consider this scenario: A frustrated customer calls your support line and says, "Great, just great" in a sarcastic tone. The ASR system dutifully transcribes this as "Great, just great" but the sarcasm, the frustration, the emotional context that would guide a human's response is gone. Your LLM sees only positive words and responds accordingly, potentially escalating the situation.

The Latency Trap

Component-based systems face an insurmountable latency problem. Each step in the pipeline (ASR, LLM inference, and TTS) adds delay. But the real killer isn't just additive latency; it's the inability to overlap processing intelligently.

In natural conversation, humans begin formulating responses before the other person finishes speaking. We use predictive processing, context, and conversational cues to prepare our replies. Component systems can't do this because they must wait for complete ASR transcription before the LLM can begin processing.

The Scale Problem

As voice AI applications scale, component-based architectures become increasingly brittle. Each component represents a potential failure point, and the complexity of orchestrating multiple services grows exponentially with volume.

More critically, each service in the chain introduces its own rate limits and scaling characteristics. Your ASR provider might handle 1000 concurrent requests, your LLM service might support 500, and your TTS provider might cap at 200. Your system's capacity is limited by the weakest link, and managing this becomes a nightmare at scale.

Companies building serious voice AI applications are discovering this the hard way. Engineering teams often spend months maintaining their own component-based systems before making the switch to unified architectures. The results consistently show dramatic improvements in deployment time and cost reductions.

The Case for Unified Speech-to-Speech Models

Instead of chaining together three brittle components, speech-to-speech systems process and generate audio directly. The entire conversation remains in the acoustic domain. This preserves context, enables predictive response timing, and eliminates cascading latency.

It also better mirrors how humans actually communicate: listening and speaking in overlapping, fluid turns—not waiting for transcripts.

Ultravox isn't just theorizing this shift. Our platform already replaces the ASR+LLM stack with a unified speech understanding model. Rather than transcribe, it interprets audio directly to produce high-quality, context-aware responses.

We’re not all the way to full speech-to-speech yet, but we're on that path. And the results already show major gains:

• Word Error Rate: Ultravox achieved 16.31% WER compared to GPT-4o's 24% and Whisper-large's 17.23%

• Response Quality: Across five evaluation criteria (sanity, helpfulness, relevance, correctness, completeness), Ultravox scored higher than all component-based systems

• Real-world Conditions: The performance gap widened significantly in noisy environments and telephony conditions

One of our customers, a fast-growing AI sales platform, saw conversions jump 37% overnight by switching to Ultravox. The gains were so outsized that OpenAI's voice team came knocking, assuming they had built a proprietary model from scratch. They hadn’t. They'd just chosen better architecture by building on Ultravox.

This is more than a performance edge. It’s proof that speech-to-speech isn’t just competitive, it’s categorically different.

The Path Forward

The future of voice AI isn't about better components; it's about unified models that understand and generate speech natively. This requires solving several technical challenges:

Alignment: How do you align speech understanding with the text-based knowledge embedded in existing LLMs? Our approach uses knowledge distillation, training the speech model to match an LLM's text-based responses while preserving acoustic context.

Multimodal Learning: Speech-to-speech models must learn from both acoustic and linguistic data. This means developing training objectives that capture both the semantic content and the paralinguistic features that make communication natural.

Production Engineering: Unified models require different infrastructure patterns than component systems. Single-model inference is simpler to scale but requires rethinking how you handle concurrent requests, caching, and resource management.

Why This Matters for Your Business

If you're building voice AI applications, the choice isn't just technical, it's strategic. Component-based systems might seem like the safer choice because the technologies are mature and well-documented. But this is exactly the kind of thinking that leads to technological lock-in because the ASR→LLM→TTS pipelines weren’t built for the realities of streaming audio, messy input, and human-level fluency.

If you’re building today, this is your opportunity to make the leap—before your competitors do.

Ultravox is your bridge to the future: a platform that understands speech natively and improves as the architecture evolves. The companies adopting unified models now aren’t just getting better performance. They’re building a moat that pipelines can’t cross.

The era of component stacks is ending. The future speaks speech-to-speech.

First off, we’re excited to announce that Fixie.ai is now officially Ultravox AI, the voice-native AI platform built for the messy, high-stakes conversations enterprises have with real customers. We started Fixie.ai when there was no ChatGPT, agents weren’t yet a buzzword, and the average person had never heard the term “LLM”. Ultravox has been our full time focus for the past couple of years, so we’re excited to fully step into our new name.

Alongside with our new brand, we’re shipping a number of major improvements to the Ultravox platform today:

• Ultravox v0.6: Our sixth-generation speech-native model, trained specifically around improving speech understanding in difficult and noisy conditions. The Ultravox model remains the leader in speech understanding.

• New model offerings: In addition to Llama 3.3, we’ve trained two new models that are available today in the platform: Qwen3 (from Alibaba) and Gemma 3 (from Google). Qwen and Gemma are some of the best models available in open source, and we’re thrilled to be able to bring them to the platform. They’re easy to prompt, great at instruction following, and excel and tool usage.

• Unlimited Concurrency: We’re removing hard concurrency caps on all of our paid plans. We’ve spent the last few months designing an auto-scaling infrastructure model that kills the “$10 per line” tax you see in legacy pricing tables.

• A new pricing ladder that lets teams graduate from free tinkering to enterprise-grade SLAs without guessing how many “concurrent calls” they’ll need next quarter.

Voice AI’s Frankenstack Problem

Walk into any voice AI demo and you'll see the same architecture: Speech to text (ASR) → LLM → Text-to-Speech (TTS). It's the obvious approach. It's also fundamentally broken:

• Latency piles up. The polite pause you hear between turns isn’t the bot “thinking.” It’s the pipeline waiting for every component to finish. Your AI forgets the emotion in someone's voice the moment it becomes text. Each component introduces failure points. One service goes down, your entire voice experience dies.

• Scalability hits a wall. These various component providers weren’t designed for real-time, so as you try to scale, all of the warts and failure points emerge quickly.

• Understanding is always diminished. Real conversation is not about just the words we say but how we say them. Human brains don’t first convert speech into text and then make sense of it. Getting to human-like conversation requires all of the context making its way to the model, something that is fundamentally unfixable in the component stack.

When conversations are short demos in perfect audio, you can hide these flaws. When you’re handling thousands of real customers across phone lines, Bluetooth headsets, and multilingual chaos, the Frankenstein stack breaks.

A Native-Voice Architecture & Why We Bet on It Early

Ultravox cofounder and Chief Research Officer Zhongqiang (ZQ) Huang, whose PhD work blends cognitive linguistics and speech processing, has always pointed to the same insight: Humans don’t think in text, then speak. We reason inside a rich, audio-and-context soup and reply in <500ms.

Our question was simple: what if the model treated speech as a first-class input instead of a codec to translate away?

Instead of translating between voice and text, Ultravox makes voice a native language for AI.

Ultravox v0.6: Speech-to-Speech Intelligence

Speech-Native Models: Our new Ultravox v0.6 model processes speech directly. No translation layer. No intelligence loss. It understands not just your words, but your tone, your timing, your intent—just like humans do.

Contextual Understanding: v0.6 uses previous conversation context to improve its predictions. If you're in India talking about food delivery and mention "Zomato," it doesn't think you said "tomato" with an accent. It remembers this is a conversation about delivery apps and gets it right.

Real-World Robustness at +30% accuracy: Unlike models trained in quiet labs, Ultravox is built for the chaos of actual human conversation. Tested against the noises of a cafe — think coffee grinders, multiple overlapping speakers, background sirens — it handles the mess because that's where real conversations happen.

Sub-500ms Response Time: Because we're not translating between modalities, we can respond in under 500 milliseconds. That's fast enough to feel natural, even when someone interrupts or changes topics mid-sentence.

We still give developers the textual transcript when you need it—think analytics, redaction, or CRM storage—but the core reasoning loop never loses fidelity.

Preview Variants: Gemma 3 & Qwen 3

Component stacks chase the next proprietary model upgrade. We built a framework to absorb new foundation models quickly and endow them with speech skills. Today we’re opening developer-preview endpoints for Google’s Gemma 3 and Alibaba’s Qwen 3.

What This Architecture Enables: Unlimited Concurrency

Here's where it gets interesting. Because we own the entire stack—from the speech models to the GPU orchestration—we can do things that component-based platforms simply can't. We can treat capacity as an elastic pool, not a line item.

Put another way: Ultravox is designed from the ground-up for scale. Whether it's one call per day or 500,000, we're ready to scale with you.

Why This Matters Now

Customers are tired of IVR trees. They expect an expert who answers like a person and never makes them repeat a 16-digit order number. Enterprises are tired of POCs. They need bots that survive weekend spikes, regional accents, and compliance audits.

By making voice native and scalability intrinsic, Ultravox turns a “cool demo” tech into a production system that:

1. Keeps full intelligence even when audio is messy.

2. Responds at human cadence (< 500 ms) so people don’t hang up.

3. Scales on demand without CFO-scary line charges.

That’s why we’re comfortable planting our flag today:

Ultravox is the first voice-AI stack built for real-world, revenue-bearing conversations — no Frankensteck required.

What's Next

Over the next few weeks we’ll be releasing a variety of new features from state-of-the-art noise cancellation to new tools to help you write (and evaluate!) high-quality prompts that are designed to scale

Get Started

Start building on Ultravox for free today at https://app.ultravox.ai, or request a demo and talk to someone from our sales team.

Last November, we introduced Ultravox v0.4.1, an open-weight speech language model designed for real-time Voice AI. While it was a strong performer among open models, proprietary models still held an edge in key benchmarks.

With Ultravox v0.5, we’ve closed that gap. This latest release brings significant improvements in speech understanding, multilingual support, and real-world adaptability. It now outperforms OpenAI’s GPT-4o Realtime and Google’s Gemini 1.5 Flash on key benchmarks for speech understanding while also maintaining the flexibility and transparency of an open-weight model. [1]

As always, the weights are available on Hugging Face.

Leading in Speech Understanding

Our goal with Ultravox has always been to give models the ability to understand natural, real-world audio without harming general reasoning or instruction following capabilities. We primarily measure our progress on two key benchmarks:

• CoVoST-2 – Measures the accuracy of speech-to-text translation across multiple languages, used as a proxy for general speech understanding capabilities

• Big Bench Audio – Evaluates general reasoning capabilities based on speech input

Across both evaluations, Ultravox v0.5 shows clear improvements over proprietary models:

Speech Translation Performance (CoVoST-2 BLEU Score) [2]

Speech-Based Reasoning Performance (Big Bench Audio Score) [3]

These improvements make Ultravox v0.5 more capable of handling complex, real-world voice interactions, with fewer misunderstandings and stronger reasoning capabilities — leading to more reliable AI-driven voice assistants.

How v0.5 Improves on v0.4.1

In addition to outperforming proprietary models, Ultravox v0.5 introduces major improvements over its previous version:

• 60% improvement in transcription accuracy, with lower word error rates (WER) across 82 evaluation sets from LibriSpeech, CommonVoice, and Fleurs.

• 18% improvement in speech-based web question answering, particularly in handling named entities and fine-grained speech details.

• Expanded language support from 15 to 42 languages, making it significantly more accessible for global applications.

Real-World Applications

While benchmark scores are useful, the real value comes from how these improvements translate into practical applications. Ultravox v0.5 makes a difference in several key areas:

• Live multilingual conversations – Businesses can process real-time speech in 42 languages without needing to know the user’s preferred language at the start of the conversation. This makes it a great choice for companies that appeal to a global audience.

• AI-powered customer support – More accurate recognition of names, commands, and intent leads to better AI-driven interactions. Ultravox can be further fine-tuned for industry specific applications to achieve even stronger performance.

Thousands of customers already built on Ultravox to handle hundreds of thousands of real-world customer interactions across customer support, inbound and outbound call handling, and AI voice assistants.

Expanded Language Support

Ultravox v0.5 significantly broadens its multilingual capabilities, increasing supported languages from 15 to 42. [4] Unlike traditional models that require pre-selecting a language for accurate recognition, Ultravox v0.5 can seamlessly switch between them in real time. Current supported languages are:

Arabic, Belarusian, Bengali, Bulgarian, Chinese, Czech, Danish, Dutch, English, Estonian, Finnish, French, Galician, Georgian, German, Greek, Hindi, Hungarian, Italian, Japanese, Latvian, Lithuanian, Macedonian, Marathi, Persian, Polish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swahili, Swedish, Tamil, Thai, Turkish, Ukrainian, Urdu, Vietnamese, Welsh.

Moving Beyond Cascaded Systems

Ultravox works by adapting strong text-based foundation models (e.g., Meta’s Llama 3.3 andDeepseek’s V3 and R1) and training them to natively understand human speech while maintaining their general language reasoning capabilities. This approach ensures that improvements in speech processing do not degrade the model’s ability to follow instructions or perform text-based reasoning tasks.

Unlike traditional cascaded systems (ASR → LLM → TTS), Ultravox v0.5 processes speech directly, enabling stronger contextual modeling while eliminating cumulative errors introduced by multi-stage processing. Early results indicate that Ultravox v0.5 outperforms cascaded approaches in difficult real-world conditions, such as noisy environments and low-quality microphone inputs. We’ll be publishing a more detailed report on this soon.

Start building on Ultravox

Ultravox v0.5 is available today through Ultravox Realtime, our managed service for building and scaling real-time Voice AI applications.

• Scales to thousands of concurrent calls

• Industry-low pricing: $0.05 per minute

• 30 free minutes to get started

Try it now: https://demo.ultravox.ai

Start building: https://app.ultravox.ai

Looking Ahead

We remain focused on our goal of building real-time, human-level voice AI that can handle the complexity of natural communication. Ultravox v0.5 is a major step forward, but we’re already exploring new architectures and techniques that will define the next generation of voice AI in both understanding and generation.

Ultravox Realtime continues to evolve, and we’re seeing strong adoption from companies around the world. We look forward to sharing more updates soon.

Join Us

Developers — Start building real-time voice AI today with Ultravox Realtime. Check out our docs, example code, and active community on Discord. 

We're Hiring — Interested in building the future of Voice AI? We're hiring. See our open positions.

—

Footnotes

1. Ultravox v0.5 benchmarked against OpenAI GPT-4o Realtime (2024-12-17) and Google Gemini Flash 1.5 (002). Model weights are available on Hugging Face.

2. CoVoST-2 BLEU scores are reported as the average across the following selected language pairs on the test set for audio samples under 30 seconds: English to Arabic, Catalan, and German; and to English from Spanish, Russian, Swedish, Turkish, and Chinese. Evaluated using Fixie AI’s evals.

3. Big Bench Audio is a benchmark released by Artificial Analysis, designed to evaluate the reasoning capabilities of audio-language models. Evaluated using Fixie AI’s evals.

4.Training data availability varies across languages, which may affect performance. We welcome feedback and contributions of training data to further enhance Ultravox.

Today we’re thrilled to introducing Ultravox v0.4.1, a family of multi-modal, open-source models trained specifically for enabling real-time conversations with AI. We’re also releasing Ultravox Realtime, a managed service that builds on top of our open-source foundations to integrate real time AI voice conversations into applications.

Unlike most voice AI systems, Ultravox does not rely on a separate automatic speech recognition (ASR) stage. Rather, the model consumes speech directly in the form of embeddings. This is the first step in enabling truly natural, fluent conversations with AI. We encourage you to try our demo to see Ultravox in action.

Ultravox shows speech understanding capabilities that are approaching proprietary solutions like OpenAI’s GPT-4o and are markedly better than other open source options. Our primary method of evaluation is speech translation, measured by BLEU, as a proxy or general instruction-following capability (higher numbers are better)¹:

Ultravox can be trained against any open-weight model. We’ve trained versions of Ultravox on Llama 3.1 8B & 70B and Mistral NeMo. Model training code is available on Github and the weights are on HuggingFace. For training your own version of Ultravox on another model or different data sets, see this section of the README.

Ultravox Realtime

In addition to the core model work, we’re also opening access to Ultravox Realtime, a set of managed APIs that make building on top of Ultravox fast and easy. When compared with OpenAI Realtime, Ultravox sees comparable latency performance.

Ultravox Realtime has built in support for voices, tool calling, telephony, and many of the other critical pieces necessary for building powerful Voice agents. SDKs are available for most major platforms and you can get started by signing up today at https://app.ultravox.ai. We’re including 30 minutes of free time to get started, after which it's only $.05/min. This makes it considerably cheaper than alternative offerings.

Ultravox Realtime builds on top of all the work that we’ve already made open source through the Ultravox repo itself and the vLLM project. Additionally, because we’re built on top of open-weight models, Ultravox Realtime can be easily customized and deployed directly onto your own infrastructure.

How Ultravox Works

The majority of AI voice implementations today rely on pipelining together speech recognition (ASR), model inference, and text to speech (TTS). While this system works for basic use cases, it fails to scale to more complex situations (e.g., background speakers, noisy environments, group conversations, etc) or where the dialogue is less formulaic (i.e., conversations without very clear turn taking). Additionally, the pipeline system introduces latency that makes natural-feeling conversation nearly impossible.

These are all problems that we’ve experienced first-hand. Our experiments with real-time AI started over a year and a half ago when we built Fixie Voice, our first attempt at enabling real-time conversation with LLMs. Despite being considerably better than classic voice-based assistants like Siri or Alexa, the conversations still felt “fake.” It was clear that the model didn’t really understand dialogue, and it could easily become confused by very common scenarios in the real world such as background speakers or natural pauses in the thought process.

The reason for this is relatively simple: pipeline systems suffer from high information loss. The ASR process strips away all context, leaving only text behind. Communication is not just about the words that we say, but how we say them and in what context. This includes emotion, intonation, tenor, and other paralinguistic signals.

Ultravox is designed to ultimately address the shortcomings of the pipeline system by training a unified model with comprehensive speech and language understanding. By giving the model more context, we can better leverage the full power of the LLM to understand what the user is trying to say.

For now, Ultravox continues to output text, but future versions of the model will emit speech directly. We’ve chosen to focus on the speech understanding problem first, as we think it’s the biggest barrier to natural-feeling interactions.

To be clear, even though Ultravox has competitive speech understanding capabilities, there is still a lot of work to be done. We outline a roadmap below that shares our intended path towards achieving truly natural communication with AI.

Roadmap to Human-level Voice AI

Our belief is simple: We think useful, productive, and accessible AGI will require models that can operate in the fast-paced, ambiguous world of natural human communication. Whether it’s support agents on the phone, AI employees joining critical meetings, or humanoid robots in the home – AI will never reach its full potential until it can cross the chasm into the “real” world.

We’re sharing our internal framework for achieving what we see as human-level voice AI.

Level 1: Assistant

Assistants are basic systems that alternate between listening and speaking modes. They are designed to provide simple information and execute basic tasks, similar to legacy voice assistants like Siri, Alexa, and Google Assistant. They are unable to carry context between turns (i.e, they are not conversational).

Level 2: Conversationalist

Conversationalists are full-duplex systems built around Large Language Models (LLMs) with the ability to engage in dialogue. Naive Conversationalists rely on separate components like Voice Activity Detection (VAD), Automatic Speech Recognition (ASR), and Text-to-Speech (TTS), which can limit the smoothness and natural feel of conversations due to component inefficiencies. In contrast, Natural Conversationalists use an integrated, end-to-end approach that enables them to understand and generate speech directly, offering seamless, expressive, and engaging one-on-one interactions.

Note: Ultravox has surpassed Naive Conversationalist, but is not yet what we would define as an “Natural” conversationalist. We think this is true for GPT-4o Realtime as well.

Level 3: Collaborator

Collaborators extend conversational AI capabilities into group settings, enabling the AI to interact effectively with multiple speakers. These systems can manage social dynamics and add significant value to professional environments and real-life applications. Collaborators are active and valuable members of a team, contributing effectively to collaborative tasks and facilitating productive teamwork.

Level 4:  Empathic Communicator 

Empathic Communicators have a deep understanding of nuances such as emotion, tone, and social context. They adapt their speech to create natural, engaging, and empathetic interactions, making them ideal for roles that require high emotional intelligence and the ability to form meaningful connections in both personal and professional settings.

Level 5: Organizer

Organizer represents the pinnacle of conversational AI, equipped to guide, influence, and inspire in complex interactions. They demonstrate advanced strategic thinking, manage group dynamics effortlessly, and deliver communication with charisma and authority.

Join Us

If you’re excited by what we’re trying to build, come join us! We’re actively hiring for engineers and research scientists. More here.

—

¹ Evaluation was performed using the following prompt: f"Please translate the text to {{language}}. Your response should only include the {{language}} translation, without any additional words:\n\n{AUDIO_PLACEHOLDER}"