Voice Agents: A New Threat?

Sumanyu Sharma, the founder and CEO of Hamming AI, previously held a significant role at Citizen, a New York-based crime reporting application. In that capacity, he was responsible for monitoring thousands of hours of police radio data and managing millions of alerts across major U.S. cities, giving him extensive experience with real-world incident management.

Sharma's work at Citizen involved constant vigilance over diverse, decentralized incidents, which now informs his perspective on the centralized risks posed by AI voice agents.

Sharma explains his heightened anxiety regarding voice agents as they transition from isolated demos and personal computers to large-scale production environments. While this advancement should be exciting, the sheer scale at which these agents will operate introduces new levels of risk not seen in physical crime reporting.

The contrast between the localized nature of crime reporting and the massive, interconnected deployments of digital voice agents underpins his concern about their reliability.

The rapid advancement in voice agent capabilities is primarily due to significant improvements in underlying infrastructure and orchestration layers, accelerating the speed at which voice experiences can be built. This foundational progress enables developers to create sophisticated voice applications more efficiently than ever before.

The industry is also seeing the development of hybrid architectures that combine different technologies to achieve both low latency and high reliability, pushing the boundaries of what voice agents can achieve in real-time interactions.

Despite advancements in speed and infrastructure, reliability continues to be the primary challenge preventing the widespread deployment of most voice agents at scale. While voice agents are often designed to sound natural and human-like, they frequently provide incorrect or unhelpful information, leading to user frustration.

The loss of user trust, when agents fail to perform as expected, is identified as a critical business consequence that can undermine adoption and investment in voice agent technology.

Sharma illustrates the tangible costs of AI failures with a personal example: a failed medical appointment booking that, while minor in this instance, could have severe repercussions if it were for a critical procedure or affected vulnerable individuals like grandparents. The consequences of such failures can range from mere time wastage to life-critical health risks, underscoring the serious implications of unreliable AI.

He emphasizes that scaling these failure modes across millions of users or critical services leads to profoundly high societal and personal costs, transforming individual glitches into widespread systemic problems.

Sharma anticipates that trillions of annual calls will soon shift to voice agents, vastly expanding their operational footprint. Even with an optimistic 1% error rate, this transition would result in 10 billion incidents per year, indicating the immense scale of potential failures.

In practice, he notes that current monitoring of 10,000 agents reveals error rates closer to 10%, meaning the real-world impact of unreliable voice agents could be significantly higher than theoretical projections.

Sharma categorizes voice agent issues by severity, drawing a parallel with real-world crime. Low-impact annoyances, such as repetitive responses, are likened to minor incidents like trash fires—inconvenient but not harmful. In contrast, high-impact risks include critical safety concerns, such as an agent mishandling a user's allergy information, mirroring severe events like mass shootings.

He emphasizes that the mass deployment of voice agents in settings like fast-food drive-thrus significantly increases the potential for danger, as a widespread failure could affect countless individuals simultaneously.

Sharma points out a crucial difference between crime and voice agent deployments: crime is typically hyper-local and decentralized, meaning incidents are isolated in specific areas. Voice agents, however, are far more centralized, operating on shared infrastructure and common codebases.

This centralization implies that a single change, such as a minor prompt adjustment or an architectural update, can have massive downstream implications, potentially impacting millions of users simultaneously across an entire system.

The second step involves prioritizing these issues based on their impact size, focusing on those that cause the most significant disruptions.

Next, teams must understand how to effectively fix the identified problems, followed by executing those fixes. The final steps include monitoring whether the changes actually worked and if they introduced any regressions elsewhere in the system, with continuous production monitoring essential for sustained reliability.

Most teams begin monitoring conversations by manually listening to calls, a crucial step for developing intuition about user interactions. While this initial manual approach is effective for gaining deep insights, it quickly becomes unscalable as the volume of calls increases.

Teams often transition to using spreadsheets or rubrics to organize known problems, but for further scalability, investing in evaluation products becomes necessary. Sharma notes that while LLM-as-a-judge tools can compute scores, they often struggle to provide novel insights into complex or previously unseen issues.

A critical example in financial services involves a voice agent designed to help users freeze their credit cards; if this essential function fails, it constitutes a massive and unacceptable systemic failure.

Such failures can have profound consequences, potentially leading to financial fraud or significant user distress, emphasizing the importance of rigorous attention to these high-stakes functionalities.

Sharma advocates for a more robust AI testing approach that moves beyond simply repeating the same test case. A better method involves keeping the core intent the same while varying wordings, patterns, accents, and speaking styles to ensure broader coverage.

Additionally, incorporating one or more new intents into the testing mix provides teams with much greater confidence that their changes are net positive and do not introduce unintended negative consequences, thus ensuring more comprehensive and effective updates.

Sharma highlights the limitations of synthetic pre-deployment settings, noting that certain fixes and hypotheses are very difficult to test adequately in such controlled environments. Real-world interactions, particularly the critical first 5 seconds of outbound calls, offer nuanced data that simulations cannot fully replicate.

Factors like vocal quality and subtle human behaviors significantly impact agent performance, making A/B testing essential for gathering accurate, real-life performance data and verifying the effectiveness of changes in a live environment.

Sharma expresses concerns that advanced voice agents, with their enhanced capabilities, create a larger attack surface for malicious actors. If these agents can extract sensitive information like trade secrets, they can certainly be leveraged to seduce individuals into revealing protected health information (PHI) and personally identifiable information (PII).

The inherent trade-off between an agent's capability and its security means that more natural, human-sounding speech can increase the success rates for bad actors. This raises the alarming prospect of weaponizing agents to target both humans and other AI systems, escalating the risks of data breaches and exploitation.

Sharma's team conducted a red teaming product test in April to validate the hypothesis that AI vulnerabilities are not merely theoretical. The results indicated a significant rate of compromise, demonstrating the ability to breach one in five agents during testing.

These tests confirmed successful breaches in critical sectors such as financial services and healthcare, revealing that even verification protocols could be bypassed. This practical demonstration underscores the urgent need for robust security measures in AI voice agent deployment.

This defense begins with a deep investment in pre-deployment testing, encompassing both text-to-text and voice-to-voice interactions to catch vulnerabilities before launch.

Crucially, a comprehensive monitoring system of all types is essential, incorporating per-call scoring and ongoing manual evaluations. For high-stakes environments, 24/7 red teaming is necessary to continuously challenge and secure the voice agent systems against evolving threats.

While voice agents offer a significantly improved user experience compared to traditional IVRs or lengthy hold queues, this advancement is a double-edged sword that could lead to a sharp rise in security incidents. Bad actors are actively exploiting the vulnerabilities inherent in these human-like agents, making them more dangerous than their clunkier predecessors.

The volume of AI-related security incidents is projected to increase dramatically, with Sharma personally convinced that voice agents represent a more immediate and pervasive threat than traditional crime, given their potential for widespread impact and exploitation.