Data Mining Fuels Agent Improvement

Moving from basic LLM usage toward a structured recipe for trace-based improvement, Vivek Trivedy identifies the goal of agent development as a data mining challenge.

The process transitions from basic LLM utilization to a structured method for improvement based on agent traces. The talk aims to clarify the complexities of data mining within the realm of autonomous agent development.

The first step in building a successful agent involves deploying it into real-world environments to gather operational feedback. This initial deployment allows the agent to interact and generate observable behaviors.

Following deployment, the second step is to collect extensive traces, which include tool calls, API interactions, and Command Line Interface outputs. The third step involves performing data mining on this stored trace data, and finally, data-driven experiments are conducted to validate adjustments in prompts, tools, or orchestration.

Agents must reflect on their past traces to update their knowledge and definitions effectively.

A robust feedback loop from the environment is essential for an agent to perform continual learning, enabling it to adapt and improve its performance over time.

Since the advent of ChatGPT, the industry has increasingly prioritized autonomy over determinism, making traditional code analysis insufficient for understanding agent behavior. Agent systems now involve complex interactions of prompts, tools, hooks, and swarms that are not easily deciphered through code alone.

Domain-specific variations further complicate the prediction of prompt behavior at scale, necessitating new tools for post-hoc understanding when agents operate autonomously in diverse environments.

LangChain centralizes trace data for individual agents or across entire project suites to streamline analysis. The team then deploys specialized agents specifically to analyze the traces generated by other agents.

This automated analysis focuses on identifying qualitative outcomes, such as pinpointing moments when users experience frustration, providing actionable insights for improvement.

Tracing allows engineers to determine if agents become less effective after compaction cycles, providing crucial insights into performance over time. This approach also enables testing counterfactuals, such as comparing the performance of different models like GPT and GLM for the same task.

Traces capture granular behavioral data that accurately reflects the actual user experience, offering a detailed view of how agents perform in real-world scenarios.

Current data volumes are considered the smallest humans will ever encounter, due to the exponential increase in agent utility. The frequency of agent output is projected to escalate rapidly, from yearly to daily scales.

This monumental increase necessitates the development of new systems designed to efficiently manage and extract valuable insights from the immense volumes of data generated by agents.

Reading agent traces at scale proves prohibitively expensive due to high input token costs and the sheer volume of trace tokens involved. Furthermore, lengthy interactions, particularly with coding agents, frequently exceed the available context memory.

Engineers must therefore develop systems that treat context as an external object, enabling efficient querying rather than simple data insertion, to overcome these memory and cost challenges.

Evidence suggests that open models can achieve judging performance comparable to frontier models, but at one to two orders of magnitude lower cost.

Harness engineering, informed by trace analysis, enables smaller models to reach similar intelligence levels through improved prompting guidance. When prompt adjustments reach a performance ceiling, domain-specific fine-tuning can allow these models to surpass frontier performance on highly specialized tasks.

This strategic combination of initial high-performance validation and subsequent cost optimization highlights a path toward efficient and powerful agent development.

For very high inference workloads, running a dedicated cluster can be significantly more cost-effective than relying on per-token pricing. These clusters provide unlimited inference capacity within their provisioned hardware, eliminating concerns about individual token costs.

Additionally, this infrastructure can be spun down when not in use, further optimizing operational expenses for dynamic workloads.

LangSplat is introduced as a product designed to automate the comprehensive analysis of trace data. The tool is capable of identifying issues, searching through traces agentically, and preparing datasets for further processing.

It facilitates the generation of evaluations and human-readable feedback directly from raw trace logs, streamlining the feedback loop for agent improvement.

Distillation involves using successful traces from larger models to fine-tune smaller, more cost-effective models, essentially mimicking superior behavior.

Evaluation defines agent behavior by the specific set of evaluations they are designed to pass, ensuring adherence to desired performance metrics. In high-trust industries such as legal and medical fields, trace mining simplifies complex data for human review, integrating a human-in-the-loop for critical oversight.

These three outputs collectively enhance agent performance, optimize resource usage, and ensure reliable operation in sensitive applications.

Modern agent development can be compared to classic machine learning workflows, drawing parallels with scikit-learn logic, according to Trivedy. The central objective is to effectively fit a model and a harness to specific data, thereby ensuring the successful completion of all assigned tasks.

Key engineering efforts are focused on identifying effective 'fit functions' and meticulously curating high-quality data. While agents excel at optimizing against specific scores, continuous monitoring is crucial to prevent them from 'cheating' or finding undesirable shortcuts.

This approach emphasizes a structured, data-driven methodology to build robust and reliable agent systems.

Agents are capable of analyzing their own past traces to identify and understand failures, moving beyond simple binary pass/fail outcomes.

This capability allows them to autonomously propose experiments and implement fixes, fostering a continuous cycle of self-improvement.

Dense feedback enables agents to accurately determine the correct next step after encountering a failure, thereby significantly enhancing their ability to learn and adapt.

Harness engineering provides rapid feedback, often within minutes, allowing for quick iterative improvements in agent performance. Fine-tuning serves as a powerful method to break through performance ceilings that harness engineering alone might not overcome.

A tiered approach, described as a 'sandwich' method—harness engineering followed by fine-tuning, and then potentially more harness engineering—is recommended for achieving optimal agent performance.

This involves the continuous adjustment of agent systems in response to real-world actions and outcomes.

The framework emphasizes the collection of observational data from agents, the iterative updating of harnesses, and the refinement of agent memory to support ongoing improvement.

Agents cannot rely on simple append-only logs for long-term memory, particularly as operations scale over years or even decades. The need for memory efficiency becomes paramount to sustain performance and learning over extended periods.

Scaling sleep-time compute is crucial for updating the agent's state using historical traces, allowing for efficient processing and integration of past experiences into current knowledge.