Smart Token Use: Anthropic's Agent Design

Anthropic designed a financial analysis benchmark to emulate the expertise of human analysts. The team created a series of complex tasks in financial analysis with the goal of replicating an expert human's performance in real-world scenarios. Initial testing focused on a 'control' group whose strategy was solely on execution.

The initial 'one-shot' execution strategy achieved a mere 15% accuracy. In unconstrained environments, models naturally adopted varying token budgets for different tasks. More complex strategies, such as 'Dreaming,' showed improved performance but required substantially more tokens, with some instances consuming up to 600,000 tokens.

To truly assess the impact of varying job assignments, Anthropic engineers fixed the maximum token budget at 600,000 tokens across all strategies. With this increased budget, standard execution accuracy improved significantly from 15% to 76%. However, strategies incorporating advising and grading roles pushed performance closer to 90%. This fixed-budget approach successfully demonstrated that even with identical token allowances, strategies like 'Advise' yielded superior outcomes, achieving 89% accuracy compared to 'Execute' at 76%, thus isolating the 'alpha' generated by strategic role assignments.

In professional domains like financial analysis, 100% accuracy is often non-negotiable for tasks such as calculating profit and loss statements. An 80% accurate result is deemed a failure because human experts must still manually recompute the entire task. This led to a shift in evaluation criteria, where only fully accurate answers are considered a 'pass' in real-world business scenarios.

Under the stringent '100% accuracy' criteria, execute-only strategies achieved a pass rate of 42%. More complex strategies demonstrated better results, reaching up to a 75% pass rate. This new evaluation framework shifted the focus toward minimizing the token cost required to consistently achieve a perfectly accurate answer.

The execution strategy, on average, requires approximately three runs to yield a perfect answer. Given a fixed budget of 600,000 tokens per individual run, this means the total token expenditure to achieve one perfectly accurate answer using the execution strategy can climb to 1.8 million tokens.

Strategies involving advising and grading proved to be more token-efficient than those relying solely on execution. Businesses face a choice: prioritize token efficiency for cost-conscious tasks, making advising an ideal strategy, or maximize reliability for critical operations where grading and dreaming strategies are preferred. This decision hinges on the specific business objectives and the tolerance for error.

The core takeaway from Anthropic's research is that tokens are not fungible commodities. Instead of simply increasing budget for brute-force execution, strategically assigning different jobs to tokens within a fixed budget leads to significantly better outcomes. This paradigm shift encourages a move away from mere volume toward intelligent task management and diversified token roles.

Anthropic's Claude Managed Agents provides an architectural framework to orchestrate individual agents effectively. This meta-harness level coordination allows for the seamless integration of execution, advising, and grading roles. The platform offers out-of-the-box support for dreaming and outcome evaluation. This structured approach simplifies the construction of complex feedback loops once the fundamental primitives are established, enabling the creation of large-scale architectures for diverse and dynamic problems.

Anthropic's long-term goal is to enhance its models and platform to dynamically construct these sophisticated strategies autonomously for users. In the interim, the company encourages users to continue combining primitives manually to build effective solutions.