Mousepower: Measuring AI Agent Efficiency

This framework aims to provide a clearer understanding of an agent's performance and value.

He stated, "today I'll be talking about mouse power and this is a talk about measuring agents through mental models." The core idea revolves around establishing a new metric that helps users assess whether the computational resources, particularly tokens, are being used efficiently and effectively by agents.

He explained that agents can assist in design explorations or other peripheral tasks while a user's primary focus is elsewhere.

He noted, "while my active attention is focusing on one thing... I still want to make some progress on peripheral tasks." This capability often leads users to run multiple agents simultaneously, allowing for progress on various fronts without direct human intervention.

While initially engaging, the excitement often wanes when the bill arrives, prompting questions about the actual value derived from the tokens spent.

Piras said, "it's a lot of fun of course until you get the bill and then you start to wonder was it all worth it, right?" This sentiment underscores a common challenge where users burn through token budgets without a clear understanding of whether agent efficiency was maximized or if the investment was justified.

Yutori focuses on building computer use models, where agents learn to operate a computer similarly to a human. These models serve as a crucial bridge in environments where traditional APIs or Master Control Programs (MCPs) are unavailable.

Piras explained, "We focus on computer use models. These are models that learn to use a computer like a human would." This approach positions agents as a last resort, enabling automation and interaction in complex, unstandardized digital environments by mimicking human input and decision-making.

Despite considerable excitement surrounding agents, many users feel they are merely scratching the surface of their potential. This often leads to confusion regarding the best use cases and the trade-offs involved in their deployment.

Piras stated, "A lot of people are excited about agents, but the a phrase that comes up quite often is that they feel like they're just scratching the surface." The challenge lies in translating initial enthusiasm into tangible utility, requiring clearer guidance on optimal applications and a better understanding of their true capabilities.

He reminded the audience that their enthusiasm for new technology does not necessarily reflect the needs or understanding of the general population.

He emphasized, "everybody in this room is very biased and we're early adopters and we're very excited to explore this new technology, but it doesn't mean that we represent the people that ultimately we're going to be trying to help adopt this technology." This distinction is crucial for developing agents that are accessible and valuable to a wider audience, moving beyond the niche appeal of early enthusiasts.

James Watt, facing the challenge of introducing steam engines to a market accustomed to horse gins, needed to understand the prevailing mental models of his potential customers. His goal was to replace the standard power source for mills, which was animal-driven, with a new mechanical one.

Piras explained, "He decided that a great use case for his steam engines was trying to replace a horse gin. ... He needed to understand the mental model of these people and specifically to create a metric that would help him give some baseline of the relative improvement in efficiency." Watt recognized the cognitive dissonance users would experience, shifting from living animals to 'cold machines'. His solution was to devise a new metric—'horsepower'—to quantify the relative efficiency gains in terms that resonated with the existing understanding of power output.

The 'horsepower' metric, despite not being strictly scientific or perfectly accurate, proved highly effective as a communication tool. Its primary function was to clearly demonstrate an increase in value and efficiency, making the new technology more appealing to skeptical users.

Piras noted, "This metric was not very scientific at the time. It was not necessarily even accurate you could say. But the main thing it did was it communicated an increase in value." This legibility was sufficient to encourage adoption, as it allowed users to understand the tangible benefits in familiar terms, effectively bypassing emotional preferences for traditional methods by focusing on clear return on investment.

Maximillian Piras asserted that the inability to provide a tangible return on investment (ROI) for customers makes communicating the value of AI agents impossible. This challenge currently hinders wider industry adoption, as businesses struggle to justify the costs.

Piras stated, "If we're not able to give something that is a tangible ROI for our customers, then it's very hard for us to communicate value." The industry faces issues like misaligned incentives and premature exhaustion of token budgets, known as 'token maxing,' due to the absence of a standardized measure for agent value.

This cycle, he argued, leads to a situation where users repeatedly 'token max' themselves into austerity.

Piras explained, "Right now I think we're kind of in this doom loop where we're overspending and we're underusing. ... it's kind of this vicious cycle where we're just token maxing and ourselves into austerity." Users frequently oscillate between excessive spending and disengaging from the technology, only to return driven by FOMO. Breaking this self-defeating pattern, he suggested, requires the development of more precise performance and value metrics.

The Coinbase CEO recently shared a chart illustrating an internal shift in their approach to AI model usage. They began changing default models, reserving frontier models only for the most challenging tasks, leading to a divergence between AI spend and raw token usage.

Piras mentioned, "There was this chart floating around on X recently that the Coinbase CEO posted where they had internally started changing the defaults of what models they will start with and trying to only save the frontier models for the hardest tasks and as a result saw some good AI spend start to diverge from token usage." This change reflects a move towards outcome-based metrics rather than simply tracking token consumption, though measuring ROI effectively without clean outcomes remains difficult.

Efficiency gains are not fully realized because most time is now spent reviewing the generated code.

Piras explained, "the bottleneck is now shifted to human review where the efficiency gains from coding agents aren't quite or aren't seen yet because we spend most of the time reviewing the code." Anthropic's team also noted that despite advances in code generation, code review remains an unsolved problem, indicating a need to address verification processes to unlock true efficiency.

Noah Hine advocates for adapting these traditional standards to accommodate new paradigms introduced by AI agents.

Piras explained, "as a culture code review has a very good convergence on shared assumptions and that lets you measure things at scale when we can all kind of converge on the measurement and it becomes somewhat of a clear rubric." Code review, as a cultural practice, relies on shared assumptions to scale effectively. With agents, these assumptions must evolve to allow for scalable and consistent measurement of their work.

Just as Watt created a baseline for power, the goal is to establish a similar baseline for human-computer interaction efficiency.

Piras elaborated, "this brings me to the idea of mouse power which could be the equivalent of horsepower for the agentic age. Just as James Walt was able to show a measure of efficiency relative to the horses in the gins that were the source of power at the time. We perhaps can also figure out how do we create a baseline of efficiency." This metric aims to demonstrate how agents can measurably outperform humans on specific tasks, thus proving a tangible return on investment.

Its purpose is to provide developers with a rubric to help users verify that an agent is performing its tasks effectively.

Piras stated, "mouse power is of is never going to be a metric of course but it's more so an idea which the idea being if you're going to sell somebody an agent you also have to help them with the with the rubric of how do we actually verify that this agent is doing good work." This concept addresses the crucial challenge of determining whether the tokens spent on an agent translate into valuable and verifiable work, ensuring users can assess the true worth of their investment.

Entropy, defined as the uncertainty of a probability distribution, forms the basis for how agents are currently trained.

Piras noted, "going back to Claude Shannon's ideas about measuring entropy in information uh entropy being the uncertainty of a probability distribution and of course very much the basis of how we train agents today." Developers should consider both the inherent complexity of the task steps and the clarity of the acceptance criteria to effectively leverage information theory for agent evaluation and training.

If the verification process is too ambiguous, the efficiency gains from using an agent diminish, making it indistinguishable from manual work.

Piras emphasized, "not just can the agent perform the task but can we help somebody actually or is is there actually a clean rubric for how it's graded." He stressed that effective agents perform best when there is a clear and well-defined rubric for verifying their output, ensuring that their contributions are measurable and valuable.

Piras identified optimal agent tasks as those that resemble 'NP-style' problems, meaning they are easier to verify than to execute. This characteristic represents a sweet spot for agent utility, maximizing their efficiency and value.

Piras explained, "they kind of become the shape of an MP style problem, which means they're easier to verify than to execute. And the reason I say that is because if you can figure out a pretty repeatable p pattern for verifying their their work, you can actually just throw agents at that problem as well." By standardizing the verification process, developers can deploy additional agents to check the primary agent's output, thereby minimizing wasted tokens and ensuring the agent provides tangible value.