Unmeasurable AI Agents Cannot Be Managed

This challenge underscores the need for clearer communication and demonstration of value to a broader audience who have not yet engaged with agent technology.

New technologies frequently encounter difficulties in effectively communicating their value to the public, a problem that often boils down to a fundamental communication gap. James Watt experienced this firsthand when introducing steam engines to replace traditional 'horse gins' in mills. Despite the advanced mechanics, people were intimidated by the unfamiliar machinery and preferred the familiar and tangible power source of horses.

This metric enabled users to calibrate their potential efficiency gains, making the abstract benefits of steam power more concrete and understandable.

Communicating value becomes nearly impossible if a tangible return on investment (ROI) cannot be clearly demonstrated to customers. Across various technology sectors, there is a recurring failure to accurately calculate or effectively demonstrate the real value derived from new implementations. A concerning trend shows companies rapidly exhausting their annual token budgets for AI agents within a single quarter, raising questions about the actual value generated.

This vicious cycle involves maximizing token usage, leading to significant financial outlays, followed by burnout and a period of waiting for the fear of missing out (FOMO) to reignite further investment. Breaking this cycle necessitates the development of superior methods for measuring and demonstrating the actual value derived from AI agent deployments.

There is a growing need to transition from simply tracking token consumption to meticulously tracing token spend directly to specific outcomes. For example, a clear metric would be how many bugs were squashed or how many support resolutions were achieved using a particular token expenditure. Companies like Coinbase are already prioritizing the use of frontier models exclusively for difficult, high-impact tasks, signaling a divergence in how AI spending is measured beyond raw token usage. This shift is crucial for linking AI investments directly to tangible business objectives.

This increased coding efficiency generates a massive influx of code that requires human oversight, effectively negating some of the initial efficiency benefits. Even development teams that have successfully automated code generation still face significant challenges with the subsequent human review process, making it the primary impediment to faster software delivery.

The focus must shift from merely accelerating execution speed to achieving measurement and verification at the speed of computation itself, rather than human review cycles.

The concept of 'Mousepower' is introduced as a potential equivalent of horsepower for the agentic age, serving as an efficiency baseline for computer-based tasks. This idea reflects James Watt's historical method of comparing machinery to horse labor to establish a clear measure of efficiency. The challenge now lies in establishing a standardized unit of performance for AI agents that can be effectively compared against human interaction.

The speaker's failed experiment to use Claude for building a mouse-movement measurement device underscored the complexity in quantifying human-computer interaction against agent performance. Simple metrics like cursor velocity fail to capture the full return on investment (ROI) and value provided by AI agents.

The core idea is that when an agent is sold, a clear rubric must also be provided to verify its effectiveness and justify its token costs. The ultimate goal is to assist businesses in justifying their investment in AI agents by offering a clear and understandable assessment of the value generated.

Entropy can be effectively utilized to measure the inherent uncertainty present in tasks assigned to agents. The proposed framework maps tasks across two axes: uncertainty in the execution steps and uncertainty in the acceptance criteria, providing a structured approach to task evaluation.

Low-uncertainty tasks often prove to be more efficiently handled by simple scripts rather than complex AI agents. The predictable information flow in flight purchases contrasts sharply with the open-ended nature of creative endeavors.

If a human is required to repeat the work an agent performed merely to verify its correctness, the agent's value proposition is significantly diminished. The most effective agent tasks reside within a "sweet spot" of manageable uncertainty, where validation is clear without mirroring the complexity of the original task.

The ideal tasks for AI agents are those with sufficient complexity to be challenging, yet enough clarity to be readily validated. These tasks often resemble NP-style problems, where verification is significantly easier than the initial execution. It is recommended to build dedicated 'verification agents' that can automate the checking and validation of outputs generated by primary AI agents, thereby streamlining the overall process.