Voice Agents: From Siri to Full-Duplex AI

Gradio, a startup based in Paris, was founded by Neil Zeghidour with a core focus on voice technology. The company emerged from the non-profit research lab QTI, which has a history of developing notable voice models such as Moshi, Hibiki, and various local text-to-speech (TTS) models. This foundational research background underpins Gradium's current work in advancing voice agent capabilities.

Their offerings include text-to-speech (TTS) services, application programming interfaces (APIs), and on-device speech-to-text functionalities. The company's core strategy involves developing robust foundation models that can be applied across a diverse range of audio-related tasks, aiming to enhance the intelligence and naturalness of voice interactions.

Its architecture was notably constrained, operating without the complex reasoning capabilities of modern large language models (LLMs). Siri primarily relied on natural language understanding and basic classification rules, which limited its utility to specific, close-ended use cases, rather than open-ended dialogue.

This evolution brought improved naturalness to conversations, allowing for more open-ended dialogues. However, these early LLM-based systems often lacked true agentic capabilities and were frequently hampered by significant latency issues, impeding real-time interaction.

A live demonstration showcased a drive-thru bot, coded with Gradium's voice models, designed to offer faster and more seamless interactions than previous generations. This advancement signifies a move towards highly responsive systems that can handle complex, real-world conversational tasks with reduced latency.

Unlike earlier models that primarily responded to queries, these advanced agents can track ongoing interactions, take specific actions, and perform real-world functions such as processing an order or handling payments. This shift represents a move from basic Q&A to sophisticated reasoning and planning, empowered by tool use, making them true "voice agents" for practical applications.

Cascaded speech-to-speech systems, while improving, still suffer from higher latency compared to human conversation, making interactions feel less natural. A fundamental limitation arises from the conversion of speech to text and then back to speech. This process often leads to a loss of non-linguistic nuances such as tone and emotion, compromising the overall naturalness and expressiveness of the generated voice.

This integration allows the models to process non-linguistic information, such as tone and emotion, leading to improved naturalness in speech. Furthermore, these models have achieved substantial latency improvements, bringing interaction speeds closer to those of human conversation.

The fundamental intelligence capabilities of these speech-centric models remain more limited in direct comparison. This implies that while they excel in naturalness and latency, their capacity for complex reasoning and understanding is not yet on par with models that rely primarily on text processing.

This half-duplex model inherently fails to simulate the continuous and overlapping flow characteristic of human conversation, which includes constant feedback and simultaneous speech. The inability to naturally overlap speech makes interactions feel artificial and annoying, representing a fundamental limitation that cannot be fully resolved with sequential turn-taking systems.

During typical phone calls, humans speak over each other up to 20% of the time, creating a flexible and comfortable conversational dynamic. This constant, simultaneous exchange of information is critical for conveying empathy, confirming understanding, and maintaining a fluid dialogue, elements currently missing in most AI voice agents.

These encoders take raw audio input and compress it into a very dense, compact representation, similar to how text is embedded. This process allows large language models (LLMs) to effectively train on audio data, enabling them to understand and generate speech in a more sophisticated manner, much like they process textual information.

This architecture allows both conversational participants to be simultaneously active or inactive, replicating the natural flow of human dialogue. This approach is currently employed by technologies like Thinking Machines for their interaction models and is likely utilized in OpenAI's bidirectional models, enhancing the ability of AI to handle simultaneous speech and more complex conversational dynamics.

Full duplex models represent the pinnacle of naturalness in voice interactions, mimicking human conversation more closely than any other system. However, current iterations of these models often sacrifice intelligence, proving to be less capable than even the less natural speech-to-speech models, which themselves are less intelligent than cascaded systems. The persistent challenge lies in maintaining a high level of naturalness while simultaneously advancing the model's core intelligence, aiming to achieve both seamless conversational flow and sophisticated reasoning capabilities.

While the latest large language models (LLMs) used in voice agents are highly intelligent, they often lack naturalness in conversation. Conversely, speech-to-speech models prioritize naturalness over raw reasoning intelligence. This occurs because the fixed capacity of an LLM is consumed by processing additional modalities, inevitably leading to a reduction in its core intellectual capabilities.

One path involves scaling up unified speech-to-speech models, aiming to integrate all functionalities into a single, increasingly powerful system. The second path focuses on modularity, proposing to split models into distinct natural interfaces and separate, robust reasoning backends, allowing for specialized optimization of each component.

A superior alternative for voice AI development lies in a hybrid architectural approach, exemplified by concepts like 'Moushiraq' and 'Thinking Machine Interaction.' This strategy involves utilizing a small, potentially on-device, full-duplex speech-to-speech interface whose sole purpose is to maintain a highly natural conversation. All complex thinking, tool calling, reasoning, and other agentic capabilities are delegated to a powerful background text large language model (LLM). This decoupling allows hundreds of natural voice interfaces to query a single, asynchronous backend, optimizing both naturalness and computational efficiency.

Running gigantic, unified models for simple conversational interactions like chit-chat is economically inefficient and wasteful. Furthermore, unified models typically tie users to a specific backend provider, limiting their ability to customize or switch components, which restricts long-term optionality for developers. Updates to complex, unified speech-to-speech pipelines are also inherently slow, whereas hybrid systems allow for more agile development and independent optimization of the natural interface and reasoning backend, making them a more cost-effective and adaptable solution.