Weights & Biases ARIA Agent Achieves Self-Improvement

Aysola's focus for the presentation was the evaluation framework that enables the agent to self-reinforce and conduct research on itself.

He highlighted the significant challenges involved in building agent harnesses, particularly due to the inherent variations in tool calls and benchmarks required for robust testing.

Aysola identified that benchmarks, evaluations, and agent configurations are all covariant, meaning they are interdependent and change together. This presents a fundamental challenge in reliably assessing AI agent performance.

He emphasized the necessity of robust measurement to verify actual performance in a dynamic system, stressing the importance of measuring across both production and offline environments to address the 'sim-to-real' gap.

Weights & Biases Weave serves as an observability platform, crucial for both production and offline tracing within the ARIA agent's development.

The team leverages production traces to perform offline hill climbing, allowing the agent to autonomously resolve errors and continuously improve its performance through a self-improving flywheel.

Aysola demonstrated ARIA's capability to take a codebase, logged as a W&B artifact, and launch training jobs within their offline evaluation framework. This enables the agent to autonomously review production traces and generate new tasks for hill climbing.

The agent develops new variants of itself by transferring production traces into the offline evaluation framework, debugging issues, and benchmarking candidate solutions.

CI jobs continuously track agent performance over seven-week periods to ensure system stability.

The system systematically monitors candidate variants against the current production version, aiming to maintain and improve performance metrics across a diverse range of tasks.

The team prioritizes prompt engineering over traditional reinforcement learning due to the advanced capabilities of modern models, focusing on building specific skills for the software agent within a simulated environment.

A 'bite-wise identical' version of the agent is used in both production and simulation to ensure consistency, with a 4-hour synchronization process in place to prevent drift between research and production environments.

To facilitate seamless experimentation, both research and production codebases share the exact same logging framework.

Internal 'run eval' commands are used to generate extensive score trajectories, which enables detailed analysis of the agent's performance. The core idea is to generate a massive amount of traces to identify emergent performance patterns.

ARIA can autonomously review its own rollouts to identify successes and failures, with feedback reinforced through prompting or other adjustments to align its behavior.

Weights & Biases utilizes a model-agnostic software stack, enabling support for both core inference and foundation models. Configuration is managed through YAML definitions, facilitating rapid, parallel variations.

The primary goal is to significantly increase the volume of experiments, allowing for deeper insights into performance improvements, with a flexible sandbox allowing the agent to build its own research loops.

The evaluation process follows a Directed Acyclic Graph (DAG) pattern, starting with YAML configurations. Environments are then hydrated with production data to ensure realistic testing conditions, crucial for maintaining testing fidelity.

Resource-intensive tasks, such as simulating GPU execution, require parallelization. Runtime configurations not encoded in YAML are hot-patched into the config, and the agent operates in a production-identical environment.

Weights & Biases prioritizes robustness in measurement to effectively bridge the gap between evaluation and production performance. The ARIA agent scores itself using two patterns: normative scoring, which determines if a task was passed, and relativistic scoring.

Relativistic scoring is particularly useful for A/B testing different interaction styles, such as assessing whether an agent's approach to asking questions improves its overall performance.

Tasks are precisely defined by YAML specifications, which outline specific start and end conditions, crucial for accurate simulation. These simulations encompass both simple text instructions and complex multi-turn interactions.

Language models are employed to simulate user personas, allowing the team to test specific sequences of questions. The system currently manages 886 tasks, categorized by level, all of which are audited by the product team to maintain quality and relevance.

Production traces are essential for understanding real-world behavior patterns of the ARIA agent. The 'evaluation flywheel' principle dictates that every production success or failure is converted into a new task for the agent framework.

This continuous feedback loop allows the agent to iterate and improve by learning from both production and offline data traces.

ARIA autonomously captures live production traces and logs them into its framework. The agent then converts these production traces into regression tasks using the Weights & Biases Agent Factory (WBAF), a process demonstrated by Aysola.

During one instance, ARIA identified a missing 'weave.log' SDK call within the sandbox environment as a specific bug. The agent then performed a hill climb, testing various variants to verify the effectiveness of its proposed fix.

This autonomous process allows the agent to benchmark its performance improvements directly against previous production versions, streamlining the debugging and optimization cycle.

Engineers can now manage agent improvements directly within the platform, eliminating the need to manually write offline benchmarks in cloud code. The platform offers unified observability for both production traces and offline evaluations, streamlining the development process.

This new workflow allows developers to allocate more time to architectural design and system guardrails, rather than being bogged down by manual coding tasks. Aysola noted that he has not manually written code in eight months, attributing this efficiency to using Claude for coding.

ARIA's self-correction mechanism involves injecting specific system prompts or updating existing skills to mitigate identified SDK errors. The agent's successful rectifications are validated by comparing the performance of the candidate variant against production metrics.

The process flow involves extracting production traces, simulating scenarios, benchmarking the agent, and then defining improvement patterns. This system empowers developers to refine the nuanced 'gray area' of how agents evolve and improve over time.