Weights & Biases Unveils ARIA Agent

Sweeney brings a background in ML and reinforcement learning from Georgia Tech, having previously served as PM for Twitter's ML stack.

ARIA is designed to enhance AI and ML workflows, with a presentation outlining its capabilities, technical architecture, and best practices for productionizing such systems.

The presentation agenda was divided into three sections, starting with an overview of ARIA itself and its ability to supercharge AI and ML workflows. A live demo showcased auto-research capabilities, followed by a deep dive into how Weights & Biases and CoreWeave built ARIA and its underlying architecture.

Weights & Biases offers a suite of products, but it is primarily recognized for its capabilities in model training, inference, and its integrated 'weave stack' for managing AI projects.

Integrated directly into a standard chat interface within the Weights & Biases platform, the system autonomously managed an autoresearch experiment based on the Karpathy project during a live demonstration.

The workspace displayed over 200 training jobs with metric tracking, showing ARIA's ability to autonomously download code, set up jobs, manage GPUs, and iterate on hyperparameters.

ARIA initiated a shell call to conduct a new batch of experiments, opting for safer hyperparameter changes rather than risky architectural modifications.

Tim Sweeney demonstrated live execution of experiments via a Kubernetes cluster terminal output, emphasizing that the demo was happening in real-time.

Weights & Biases Launch connects compute clusters, enabling agent-driven jobs to manage long-running experimentation using GPUs in real-time.

Team onboarding and pattern identification were facilitated by the agent as it summarized high-performing runs and isolated key variables like batch size during the research process.

The agent identified batch size as a critical parameter lever, discovered a new promising model family, and highlighted architectural recipes that would typically take hours or days for human researchers to uncover.

It generates W&B reports, described as 'markdown files on steroids,' incorporating charts, graphics, and parameter importance plots.

The agent is custom-tuned to build and manage W&B workspaces and proprietary charts, providing a comprehensive visual understanding of experimental data.

ARIA functions as a data science companion within the W&B ecosystem, designed to complement researchers by handling operational mechanics. The system is built from the ground up to understand primitives and co-pilot research efforts.

Weights & Biases announced the release of ARIA on its iOS app, allowing users to manage tuning jobs remotely. The ultimate goal is to evolve into a fully automated end-to-end research platform.

The architecture features a web and iOS client that communicates with an API server, which then stores data in a database. This data is processed by a 'worker harness' responsible for managing tasks.

The worker harness integrates with a sandbox for Python and shell execution, connects to an LLM provider, and utilizes long-running workload tools like Weights & Biases Launch for efficient operation.

Weights & Biases Weave logs 100% of traces, enabling thorough team analysis.

This extensive logging allows for offline analysis, helping researchers identify specific behaviors and create unit tests (tasks) for model improvement.

The improvement loop involves forming hypotheses, testing candidate agents, and evaluating results to update the production model, ensuring continuous enhancement of the system.

The 'spans view' offers a visual indicator of a trace's topology, using different colors and shapes to distinguish between tool calls, LLM calls, and reasoning blocks. This allows teams to filter and review internal interactions and understand the agent's decision-making process.

This self-referential analysis allows the team to generate recommendations for improvements directly within the UI, continuously enhancing ARIA's performance and capabilities.

Weave includes an integrated LLM judge experience called 'Signals,' which runs live against production traffic. These judges track events such as 'user frustration moments' or 'low-quality responses.'

These automated markers help the research team cluster behaviors, allowing them to prioritize fixes and improvements for future iterations of the agent.

An LLM judge determines the correctness and interest level of insights generated by the models.

Additionally, a rule-based judge checks for efficiency, specifically ensuring adherence to tool call limits. Approximately 200 tasks are clustered into an evaluation suite that runs nightly, providing continuous assessment.

Weave tracks evaluation metrics over time through interactive dashboards, providing clear performance plots from nightly evaluation results to aid decision-making.

A candidate model was observed to achieve a 73% score compared to the 72% of the production model, demonstrating how these metrics provide the confidence needed to push new model versions into production.

Investing in agent-oriented observability is crucial for identifying 'behavioral bugs' that go beyond mere performance issues, ensuring a deeper understanding of agent interactions.

Tasks and evaluations should be treated as a new form of CI/CD, integrating researchers into scrum teams to streamline development and deployment processes. Human oversight remains essential to catch subtle behavioral nuances that LLM judges might miss.

It is important to avoid over-engineering memory and instead focus on providing agents with relevant business domain context and data. Regularly reviewing both the best and worst traces as a team at the end of each week is critical for continuous improvement.

The model achieved a result of 5.833, nearly matching the previous benchmark of 5.831, demonstrating its effectiveness in autonomous experimentation. The platform continues to run additional experiments overnight.