Anthropic Proposes 'Tokens Should Have Jobs'

Anthropic introduces a core thesis stating that tokens within AI systems should be assigned specific jobs rather than being treated as interchangeable resources. Katelyn Lesse emphasized that this idea forms the fundamental concept behind their research. Current agentic systems often treat tokens as fungible units, where performance improvements primarily depend on increasing the total token budget. Anthropic's proposal aims to transcend this limitation by assigning distinct, specialized roles to tokens beyond simple execution.

Angela Jiang elaborated on the current state of AI agent execution, describing it as indiscriminate. In current models, all tokens perform the same task of executing, acting as generic processors. This undifferentiated approach means that every token spent contributes uniformly to the execution process, without specialized functions. The proposed model from Anthropic seeks to diversify these roles to significantly improve overall system performance and efficiency, moving away from this monolithic execution strategy.

These strategies are designed to diversify how tokens contribute to an agent's task. Each strategy provides a specialized function to enhance the AI agent's overall capabilities.

The executor performs the primary task but can consult the adviser for guidance. This dynamic allows the executor to refine its actions and ensure correct step-by-step progress, pulling advice as needed. This strategy is particularly applicable for systems like sales agents, where an adviser could flag overdue follow-ups or identify stalling deals, thereby optimizing the agent's performance.

This method ensures that all outputs meet specific, predefined criteria. For instance, in a customer service context, a grader could assess how well an agent handles refund requests by evaluating adherence to company policy. Each time an executor attempts a task, a provisioned grader assesses its performance against the rubric, forcing refinements until the outcome is satisfactory.

In this strategy, a 'dreamer' inspects the work and transcripts generated by the executor. Any insights or learnings derived from this inspection are then written into memory, making them available for subsequent rounds of operation. This strategy is highly effective for tasks such as recruiting, where an agent can continuously improve its ability to identify suitable candidates by learning from prior interactions and outcomes.

Katelyn Lesse described the creation of a benchmark consisting of various tasks related to financial analysis. The team's goal with each task was to replicate the performance of an expert human financial analyst in a real-world setting. This benchmark served as the foundation for evaluating different token job strategies. The initial baseline measurements focused on assessing performance through simple execution, providing a starting point for comparison with more complex, role-based approaches.

Katelyn Lesse presented initial performance data, highlighting the 'Execute' strategy's efficiency and limitations. The 'Execute' strategy achieved only 15% accuracy, but it did so by spending a relatively low 39,000 tokens. In contrast, more complex strategies, while performing better, consumed significantly more tokens, with some using up to 600,000. This initial comparison revealed a trade-off between performance accuracy and token consumption, demonstrating that simple execution was token-efficient but less accurate.

To precisely determine if varying token jobs produced any 'alpha,' or distinct performance advantage, Katelyn Lesse explained that the team needed to hold the token budget constant. They fixed the maximum budget at approximately 600,000 tokens, based on the consumption of the 'Dreaming' strategy, and applied this budget uniformly across all strategies. Although performance generally increased for all strategies with a higher budget, significant gaps between the effectiveness of different strategies persisted, indicating that the chosen strategy itself played a crucial role.

With the fixed budget, the 'Execute' strategy achieved a 76% accuracy rate, while the 'Advise' strategy reached 89%. This substantial variance confirms that the choice of strategy itself provides significant performance benefits. This finding underscores the importance of diversifying token roles, rather than solely relying on increased computational power, to enhance AI agent capabilities.

Angela Jiang emphasized the stringent accuracy requirements in domains like financial analysis, where an 80% accurate result, while seemingly good on a benchmark, is practically useless. In such scenarios, any deviation from 100% accuracy forces an expert to manually recompute the entire profit and loss statement or run the task again. Therefore, for tasks of this nature, if the result is not 100% accurate, it is considered not useful. To reflect this reality, the team adjusted their evaluation criteria to a strict 'pass/fail' based solely on perfect accuracy.

In stark contrast, more complex strategies demonstrated significantly higher performance, achieving up to a 75% accuracy rate under the same rigorous conditions. This analysis underscored the necessity of specialized token roles to meet real-world accuracy demands.

When operating a business, the critical metric is the cost incurred to obtain that perfect, actionable answer. The team is now primarily focused on analyzing the average cost required to consistently achieve the 100% accuracy mark, aligning their research with practical business imperatives. This shift in focus acknowledges that perfect accuracy must also be economically viable to be truly useful in enterprise settings.