A Linguistic Framework for Voice AI

Her background in speech communication research underpins her approach to analyzing current voice AI systems. The primary goal is to provide a linguistic perspective to better understand and improve how these systems function and interact with users.

Kim recounted a personal experience where a voice bot failed to correctly identify her name, 'Midam.' The system initially guessed 'Midan' and then mispronounced her name as 'Madam,' even after she attempted to correct it by spelling it out. This incident underscored a significant flaw in current voice AI: its inability to accurately recognize and personalize interactions, which led to user annoyance.

Another frustrating interaction involved a bot interrupting Kim while she was providing an account number. The system cut her off before she could finish, then stated it couldn't find her record, indicating a failure to track user progress during the conversation. This lack of intelligent error handling and perceived empathy further highlighted the deficiencies in how current voice AI manages user input and interaction flow.

Kim stressed that human communication operates as a joint activity, where participants continuously process and update their mental models based on mutual exchanges. This constant adaptation and shared understanding are crucial for effective conversation. Therefore, voice AI systems must be architected to treat conversation similarly, rather than as a series of isolated commands, to achieve meaningful and successful interactions.

Analyzing the 'Midam' vs. 'Madam' failure through a linguistic lens, Kim identified specific shortcomings in both Speech-to-Text (STT) and Text-to-Speech (TTS) components. The STT system failed to differentiate between the 'M' and 'N' sounds in her name, while the TTS system incorrectly applied English-centric pronunciation rules to a non-English name. This interaction failure stemmed from the bot's inability to establish a shared mental model, leading to misinterpretations at fundamental linguistic levels.

Kim outlined the core components essential for human-to-human conversation that should be mirrored in voice AI systems: distinct listening and speaking channels. Effective communication within these channels requires managing sounds, words, interaction dynamics, and maintaining a shared mental model. Voice AI systems are expected to integrate these components to achieve a level of conversational competence comparable to human interaction.

A crucial 'mental model' layer is integrated to ensure the bot comprehends user intent throughout the conversation. This framework systematically organizes various engineering, linguistic, and cognitive science terms, ensuring each element has its designated place within the voice AI's architecture.

Kim emphasized that all components within a voice agent system are interdependent, not isolated. The quality of sound must align seamlessly with word choice, and interaction timing for a successful conversation. A failure in any single layer, such as poor sound quality or inappropriate word selection, will inevitably lead to a complete breakdown of the voice agent's functionality.

Unlike text-based interactions, the sounds, words, and interactions in voice communication vanish the moment they are spoken, as waveforms dissipate. The only persistent element is the user's mental model, which continuously evolves throughout the interaction. Therefore, voice AI systems must be designed to account for this fleeting nature, focusing on building and updating the mental model dynamically over the course of a call.

Optimizing voice agents requires dynamic orchestration, as conversational elements are constantly changing throughout a call. This involves selecting optimal Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) models, along with effective pre and post-processing techniques. Careful vocabulary curation, sophisticated turn-taking management, and capabilities for emotion detection and dynamic context retention are also crucial to cater to diverse users effectively.

Paying close attention to this linguistic framework yields significant business implications by reducing user frustration, task failures, live agent escalations, abandoned calls, and silent failures. To diagnose and measure the end-to-end performance of voice agents, ServiceNow developed 'Eva Bench.' This tool helps demonstrate how improved linguistic orchestration directly translates into tangible business benefits.

Effective communication is a joint activity where both parties, including humans and AI, continuously adapt to each other. Users learn and adjust their communication styles to the voice agent throughout a conversation. Therefore, voice AI systems must be designed with inherent flexibility to adapt to the evolving nature of language and changing user expectations over time.