Voice Agents Can Just Do Things

Charlie Guo of OpenAI clarified that speech is not the sole output method for voice models, emphasizing that models have evolved to support diverse design patterns.

Guo urges developers and users to expand beyond simple conversational loops, envisioning a future where voice agents can perform actions or provide visual feedback without necessarily speaking back to the user.

These modes include speech-to-speech, where agents engage in direct vocal conversation, and speech-to-action, where spoken commands trigger specific tasks or functions.

Another significant mode is event-to-speech, where agents proactively communicate information based on internal or external events. Guo noted that these interaction patterns are not mutually exclusive and frequently overlap in practical applications.

Charlie Guo acknowledged that the underlying principles are not entirely new, citing examples from previous talks and existing technologies.

Early speech-to-speech systems were seen in movie phone hotlines, allowing users to interact verbally for information. GPS navigation systems serve as a legacy example of event-to-speech interactions, providing spoken directions based on real-time location data. Today, technology combines these precedents in more innovative and powerful ways.

Language learning benefits significantly from live practice and coaching, where the ability to interpret and respond to nuances like emphasis and emotion provides powerful feedback.

Customer support experiences can be greatly enhanced, moving beyond traditional phone trees to high-quality AI agents capable of nuanced conversations. These agents can understand complex queries and provide accurate, empathetic responses, improving user satisfaction and efficiency.

Modern voice models have advanced significantly in both quality and speed, enabling low-latency dynamic translation. Charlie Guo highlighted that content can now be translated on the fly with minimal to no delay, opening up new possibilities for global communication and media consumption. This capability paves the way for future live-dubbed events at scale, making multilingual content accessible instantaneously.

He noted that models can use tools to perform actions rather than merely generating speech, a significant untapped capability in the current market.

This mode extends beyond simple commands, encompassing applications like form filling and administrative automation, where voice input can streamline otherwise tedious manual processes.

Charlie Guo observed that much of internet usage involves form completion, and there is no longer a reason why users shouldn't be able to simply speak their input.

This method allows for conversational data entry, potentially completing up to 90% of complex forms more efficiently than traditional typing. It offers a superior user experience, making administrative tasks less cumbersome and more intuitive.

Charlie Guo looks forward to a future where users can articulate their aesthetic intent via voice, even without specialized software skills.

This approach helps overcome a lack of expertise in complex software like Photoshop or Ableton. By speaking commands, creators can directly translate their taste into technical actions, democratizing access to professional-grade creative capabilities.

Charlie Guo pondered why users should interact with individual apps or terminals when models are becoming capable of performing everything a human can do on a computer.

This vision proposes talking directly to the entire computer, treating the entire desktop environment as a single conversational interface. It moves beyond the confines of traditional interfaces, allowing for a more natural and integrated user experience.

This involves utilizing existing API endpoints and React hooks as tool definitions, effectively converting software logic into model-accessible tools. It is crucial to incorporate guardrails and safety checks during this integration, ensuring voice functionality enhances, rather than replaces, fundamental UI elements.

Charlie Guo pointed out that verbal explanations are not always necessary to convey what is happening on screen with software, as visual cues can provide immediate and intuitive feedback.

Combining voice with visual elements like notifications, state changes, button color shifts, text highlighting, and cursor animations creates a richer user experience. This approach avoids over-reliance on speech alone for user updates, ensuring clarity and efficiency.

Charlie Guo described this as an escalatory path where voice acts as a final layer to grab user attention after visual notifications have failed.

This mode allows for hands-free or screen-free interaction scenarios, such as cooking, where an agent can proactively speak based on detected events. Voice becomes a crucial communication channel, escalating user engagement when visual cues are insufficient or unavailable.

Charlie Guo shared personal anecdotes of programmers who thought their careers were over due to loss of manual dexterity.

However, the advent of large language models, coding agents, and voice agents allowed these individuals to generate orders of magnitude more code than they previously could. This demonstrates the profound potential of these technologies to unlock opportunities and enhance productivity for a broader segment of the population.

The future of voice agents lies in multimodality, combining disparate interaction modes into a seamless single agent across various platforms. Charlie Guo explored the concept of a unified voice agent, particularly in real-world applications such as automotive and gaming environments.

Current in-car assistants like CarPlay exhibit limitations, often requiring separate interactions for navigation, music, or calls. A unified voice agent would integrate these functions, allowing for natural, continuous interaction without switching contexts.

In game spaces, multimodality enables characters that not only perform actions based on voice commands but also provide information verbally, creating a more immersive and interactive experience. This unified approach moves beyond siloed functionalities, offering a cohesive and intuitive user interface.

The challenge involves integrating speech-to-speech, speech-to-action, and event-to-speech capabilities into a single, intelligent system that understands context and user intent across diverse scenarios.