Your AI Agent Evolved, But Your Evals Didn't: Braintrust CTO on Adapting AI Architectures

Ameya Bhatawdekar, the Field CTO at Braintrust, introduced himself and his company. Braintrust operates as an evaluation and observability platform specifically designed for AI teams, aiming to address the complexities of AI engineering.

Bhatawdekar noted that while creating an AI demo is straightforward, scaling it to production quality presents significant challenges. This difficulty stems from the need for frequent adjustments and updates, driven by the rapid evolution of AI models, shifting user behaviors, and constantly changing data landscapes.

He pointed out that this rapid pace makes maintaining a production-ready AI application a continuous effort.

AI models are undergoing rapid advancements, incorporating new capabilities such as enhanced tool use, extended context windows, and secure code execution. Bhatawdekar stressed that these developments are not incremental improvements but rather 'step-function changes' that fundamentally alter what models can achieve.

Consequently, traditional iterative development methodologies are no longer sufficient; instead, these significant model shifts necessitate systematic re-platforming of AI systems to leverage the new capabilities effectively.

Previous AI systems were designed and built under specific assumptions regarding the limitations and constraints of older model capabilities. As new models emerge with advanced functionalities, these legacy architectures often become restrictive, preventing developers from fully utilizing the new potential.

To properly integrate and benefit from capabilities like improved tool calling, developers are frequently forced to re-architect their entire systems, rather than simply updating components.

The introduction of new model capabilities inherently creates new potential surface areas for failure within AI systems. Therefore, evaluation strategies must evolve concurrently with architectural changes, as the architecture itself is driven by updates to the underlying models.

This dynamic means that evaluations cannot remain static; they must adapt to the new complexities and potential vulnerabilities introduced by each architectural shift.

To illustrate architectural evolution, an SRE (Site Reliability Engineering) agent was introduced as a practical example. This agent possesses the ability not only to read information but also to perform actions and update systems, such as rolling back deployments or escalating issues to human operators.

Equipped with both read and write tools, this SRE agent serves as a valuable model for analyzing and demonstrating how AI architectures must adapt to new capabilities.

Approximately three years ago, a significant number of AI applications commenced with a very straightforward architecture, characterized by a single prompt leading to a single model call. This foundational setup represented the simplest approach, with one clear input feeding directly into the model and generating a single output.

This initial generation of AI systems was designed around direct interaction, where the user's query or instruction was encapsulated within a single prompt, which the model processed to produce a corresponding response without complex intermediate steps or multi-model orchestration.

While effective for basic tasks, this simple architecture highlighted the nascent stage of AI application development, where the focus was on direct input-output relationships rather than sophisticated reasoning or multi-tool integration.

As AI systems evolved to incorporate more complex graph-based architectures tailored for specific use cases, new points of failure emerged. This increased system complexity necessitated a shift in evaluation scope beyond overall orchestration.

Evaluations now must include tracking the performance of individual nodes, monitoring retry loops, and observing intricate orchestration behaviors to ensure comprehensive system reliability.

By late 2025, significant improvements in tool calling and planning capabilities rendered many static graph-based AI systems brittle and inefficient. These rigid architectures were often unable to fully leverage the new, more advanced model functionalities.

As models gained the capacity for introspection and self-correction during execution, they started to outgrow the constraints of predefined, inflexible graph structures.

The resurgence of ReAct-style loops has enabled AI models to reliably achieve user intent autonomously, even with high variance in execution paths. High variance means that the same input can lead to different processing trajectories, necessitating a new approach to evaluation.

Instead of focusing on a single evaluation outcome, the unit of assessment has shifted towards analyzing the statistical distribution of evaluations. This allows for a more robust understanding of an agent's performance across diverse and unpredictable scenarios.

To address the challenges of high variance and agentic systems, new metrics like pass@k and pass-wedge-k have become essential. Pass@k measures the system's capability, determining if a task succeeds at least once over k attempts with the same evaluation.

Conversely, pass-wedge-k assesses reliability by calculating the consistent success rate across k instances of the evaluation. These metrics provide a more comprehensive view of an AI system's performance, balancing its potential with its consistency.

There is a significant architectural shift transforming AI systems from simple models running in loops into comprehensive product systems. Modern AI applications now integrate persistent memory systems to maintain context across sessions, enhancing user experience and continuity.

Furthermore, these systems incorporate robust sandboxes that enable reliable execution of model-generated code, expanding their functionality and interaction capabilities securely. The inclusion of skill directories and symbolic instructions allows for extending model capabilities and provides structured ways for models to access and utilize specific tools or knowledge efficiently.

Continuing to use evaluation methods from previous generations of AI will inevitably result in only partial coverage of modern, complex systems. Newer architectures introduce entirely new surfaces of fragility that older metrics and testing approaches are simply not equipped to capture effectively.

These coverage gaps mean that critical vulnerabilities or performance issues in sophisticated AI systems might go unnoticed, leading to unreliable or underperforming deployments. To ensure comprehensive validation, evaluation strategies must constantly adapt to the increasing complexity and evolving capabilities of contemporary AI architectures.

Failing to update evaluation methodologies in line with architectural advancements leaves systems susceptible to unforeseen failures and prevents developers from fully understanding and optimizing their AI products.