Real-time Voice Agents Built on Modular AI Frameworks

EliseAI models its real-time voice agent architecture on the perception, planning, and control systems found in autonomous vehicles. This approach breaks down complex interactions into manageable, specialized components, ensuring robust and efficient operation.

Transcription of real-world audio signals into processable data forms the perception layer, akin to how self-driving cars interpret their environment. The planning layer, powered by a language model, processes these inputs to generate appropriate responses, while the control layer handles the conversion of text into expressive audio for seamless voice delivery.

EliseAI has developed a streaming speculative transcriber that layers a fast streaming transcriber, Flux, with a more accurate, context-heavy batch transcription layer, Scribe V2. This combination allows for immediate responses while simultaneously refining transcription accuracy.

Flux provides quick, initial transcriptions, enabling the agent to start processing user input without significant delay. Concurrently, Scribe V2 processes the same audio in batches, leveraging a broader context to correct any initial inaccuracies from Flux. This two-tiered system ensures both speed and precision in understanding user utterances.

A corrective layer actively cancels outdated streaming text as soon as more accurate, context-aware information arrives from Scribe V2. This dynamic correction mechanism ensures the agent's understanding evolves in real-time, preventing misinterpretations and improving the overall responsiveness of the system.

This allows the main agent to maintain a smooth conversational flow by assuming tool calls have already been made within its context.

Early generation triggers enable the agent to begin preparing responses even while waiting for finalized user input. This predictive processing helps to mask inherent delays, making interactions feel more immediate and natural to the user.

The agent dynamically cancels previous generation attempts when new, corrected context becomes available from the Scribe V2 layer. This ensures that the agent's response is always based on the most accurate understanding of the user's intent, preventing outdated or incorrect information from being used.

The system employs phonetic matching and context-aware error correction to resolve inconsistencies in names and dates, further enhancing accuracy. The agent only emits its final response after confirming the user's utterance has concluded, preventing interruptions or premature responses.

To optimize real-time audio generation, agents must stream audio as quickly as possible, effectively masking generation latency. This involves initiating audio playback even before the agent has finished generating the full text response.

This technique successfully hides processing delays, making the interaction feel more immediate and natural for the user. By overlapping audio generation with text generation, the system creates a smoother, more responsive conversational experience.

The system waits for a few words to be generated before triggering a cache lookup. If a match is found, the cached audio can be played almost instantaneously, bypassing the need for real-time text-to-speech synthesis.

When unique tokens cause a cache miss, the system seamlessly transitions to standard text-to-speech (TTS) synthesis. This ensures that even when pre-generated audio is unavailable, the user experience remains uninterrupted, with new audio being synthesized on the fly.

Cartesia TTS is employed to provide natural prosody for full sentences, maintaining a human-like conversational quality. The system actively suppresses any overlapping audio to ensure a seamless listening experience, preventing choppy or disjointed responses.

During this process, the agent yields the cached audio first, followed by the rest of the streaming text as it becomes available. This layered delivery ensures that the user hears a continuous, natural response without perceiving the underlying technical transitions.

A live demonstration showcased an automated call scenario involving an OBGYN appointment scheduling. The agent successfully navigated a complex interaction, including insurance verification and scheduling, demonstrating its real-world applicability and robustness.

The example effectively highlighted the practical application of streaming and caching technologies in a demanding conversational environment. The agent's ability to handle multi-step processes with natural language demonstrates the efficacy of EliseAI's architectural innovations.

The company is actively expanding its operations and presence in the Bay Area, aiming to make a tangible impact on people's lives through practical AI applications.