Custom Evals Are Key to Agent Performance

Understanding agent testing requires a precise vocabulary to differentiate between its components. An 'eval' refers to a single test of one specific behavior or task performed by an agent. A 'benchmark' is a comprehensive and repeatable collection of these evals, designed to measure performance across a suite of tasks.

The 'rubric' outlines the criteria used to grade an agent's performance, defining what constitutes success or failure for each eval. The 'trajectory' captures every action and decision an agent makes during a task, providing a detailed step-by-step log. Crucially, a 'task' is more than just a prompt; it encompasses the initial system state, available tools, documentation, and specific instructions given to the agent, creating a complete operational context.

Most prevalent public 'evals' today consist primarily of basic question-and-answer prompts, which fail to simulate real-world agent interactions. These evaluations do not challenge agents to use tools, manipulate files, or engage with actual product interfaces, leading to an oversimplified view of their capabilities.

The current testing landscape often relies on trivial prompts that bear little resemblance to authentic product usage. This gap means that existing benchmarks often overlook critical aspects of agent performance, such as their ability to handle complex workflows or interact with various product surfaces, leaving significant real-world challenges unaddressed.

Evaluating agent performance solely based on its final outcome can lead to misleading pass rates and false positives. A successful outcome does not necessarily indicate optimal or correct agent behavior if the path taken to achieve it was inefficient or flawed.

True assessment requires examining the agent's complete trajectory, including every tool call, the number of retries, latency encountered, and any bypasses made. For instance, an agent might achieve its goal by unexpectedly bypassing a target vector database and using an SQLite database instead. While the final goal is met, this deviation reveals a critical inefficiency or incorrect assumption in the agent's design or underlying model, which would be missed by output-only evaluation.

Public benchmarks like SWE-bench are designed to rank different models against each other, focusing on broad capabilities rather than product-specific performance. While valuable for general model comparison, they fail to answer crucial questions about an agent's ability to integrate with proprietary systems.

Standardized tests cannot verify an agent's effectiveness when using unique documentation, custom APIs, or specific data schemas relevant to a particular product. Real-world agent workflows frequently involve complex operational states, such as user permissions, billing processes, and specific environment configurations, which are entirely overlooked by generic benchmarks. Consequently, these public evaluations cannot provide meaningful insights into how an agent will perform within a company's actual product ecosystem, making custom evaluations indispensable for practical deployment.

Experiments were conducted using Sentry as a test case to assess the behavior of agents like Codex and Claude on basic tasks. The evaluation involved analyzing not just the final output but also the intermediate steps and tool calls the agents made.

This analysis focused on key metrics such as the total number of steps taken, the frequency of tool calls, and the token cost incurred by each agent. This granular examination helped identify crucial friction points and issues like context window truncation, which significantly impact agent efficiency and reliability in complex environments.

Agents frequently fail to process complete documentation or information due to context window truncation issues, where longer texts are silently cut off. This means the agent operates with incomplete data without any explicit error or warning.

These truncation-induced failures often remain undetected by engineering teams, as they do not manifest as crashes or overt bugs. Custom evaluations, designed to meticulously track what information the agent actually processes, are the only reliable method to uncover these hidden failure points. Without such specific testing, critical gaps in an agent's understanding and capability would persist unnoticed, undermining its overall reliability.

Instead of manual analysis, agents can be used to analyze the failure trajectories of other agents, significantly reducing the manual effort involved in building analytics. This approach leverages an MCP (Management Control Plane) to automate the review process.

The key to effective automated trajectory analysis is the consistent collection of execution traces. These traces provide the detailed, step-by-step logs that an analytical agent can then process to identify how an agent recovered from errors or where it struggled, bypassing the need for extensive human-built analytics infrastructure.

Different large language models (LLMs) exhibit distinct behavioral patterns when performing tasks. For example, Codex typically dedicates significant time to planning and deliberation before executing a task and formulating an answer. This methodical approach focuses on optimizing the initial strategy.

In contrast, Claude adopts a more iterative, trial-and-error methodology, attempting actions, evaluating their outcomes, and adjusting its approach if the initial attempt fails. These behavioral differences are reflected in performance metrics such as step duration and token consumption per tool call, highlighting the need for varied evaluation strategies.

Relying on a single trial for agent evaluation is insufficient to produce reliable results, as tests can be inherently 'flaky' due to transient factors or non-deterministic agent behavior. A single instance may not accurately represent the agent's typical performance.

The only effective solution for mitigating flaky tests is to implement recurring evaluation cycles with multiple trials. This repetitive testing normalizes results, provides a more robust dataset, and helps to distinguish between true performance issues and intermittent anomalies, ensuring the stability of the evaluation suite.

Overlooking these can lead to cascaded failures across the entire evaluation process. Robust infrastructure is required to detect these limits in real-time.

This infrastructure must include automated mechanisms to pause the evaluation process when rate limits are hit, then intelligently resume when API slots become available. Implementing multiple trials further helps to circumvent the impact of transient rate limit issues and ensures that evaluation results are not skewed by temporary service availability problems, maintaining the integrity of the benchmark.

The most significant challenge in building effective agent benchmarks, even for large companies, lies in precisely articulating the specific use cases to be tested. This foundational step often proves difficult because it requires a deep understanding of user flows and a clear definition of what constitutes successful agent interaction.

Many teams struggle to develop robust evaluation sets due to a lack of prioritization in documenting user-centric test cases. Automated tools cannot replace the subjective product knowledge and domain expertise needed to accurately map user behaviors and define success criteria. Therefore, overcoming this hurdle necessitates a concerted effort to codify real-world user scenarios, ensuring that evaluations genuinely reflect how agents will be used in practice.

Real product interactions involve complexities that mocks cannot fully replicate.

Instead, evaluations should utilize isolated test accounts pre-filled with representative dummy data. While this approach is more complex, the industry is actively developing automated data seeding and cleanup procedures to ensure test accounts are consistently prepared for repeatable evaluations, allowing for more authentic and reliable performance assessments.

Understanding these struggle points is far more valuable for product improvement than a simple success or failure label.

'Unrecovered' frictions, where the agent fails to resolve an issue, are prioritized as critical bugs that demand immediate attention. Conversely, 'recovering' frictions, where the agent eventually overcomes a hurdle, highlight areas for optimization. Analyzing why an agent triggers high documentation usage or excessive resource consumption is crucial for refining its behavior and enhancing overall product efficiency.

To ensure agents correctly select and utilize the intended tools or API endpoints, rubrics can be implemented as part of the evaluation process. These rubrics verify if the agent's actions align with predefined optimal choices.

Visual run profiling tools provide insights into which product surfaces an agent actually accesses during a task. Analyzing these usage patterns helps identify unused or redundant API tools that can be deprecated, streamlining the agent's operational environment and improving efficiency.