Microsoft Introduces FinOps for AI Agents: A New Approach to Managing Token Consumption and Costs

Many organizations struggle with the common scenario of receiving an AI bill from their agent workflows without being able to trace its exact origin. This lack of visibility makes it difficult to understand where the costs are coming from and how to manage them effectively. Compounding this issue, the current industry culture often encourages 'token maxing,' where the priority is to explore and utilize as many tokens as possible, rather than focusing on cost efficiency.

The core objective of this new approach is to transition from simply maximizing token usage to maximizing the value derived from those tokens. This shift redefines success not by how many tokens are spent, but by the tangible value generated. The central question becomes not just who spent the tokens, but what specific value was created in return for that expenditure, fundamentally altering how AI agent performance and cost-effectiveness are evaluated.

Throughout the evolution of software, each era has developed its own control surfaces to manage costs and usage. In the SaaS era, the user interface served as the primary interaction point, with controls typically manifesting as usage caps, seat limits, or tier-based policies. The subsequent cloud era introduced a pay-as-you-go model, necessitating advanced controls such as autoprovisioning and autoscaling policies to manage dynamic resource allocation. However, the emerging agentic era currently lacks similar sophisticated control planes at the critical layer where code directly interacts with models, creating a significant gap in cost management capabilities.

The lack of effective cost controls in AI agent workflows has led to alarming real-world consequences, with companies facing immense financial burdens. For instance, Uber reportedly exhausted its entire AI budget within just four months. Moreover, some companies have incurred costs in the hundreds of millions of dollars within mere days or months, primarily due to runaway agent loops that consume tokens without bound, highlighting the urgent need for a robust cost management solution.

To effectively manage AI agent costs, several first principles must be established. First, tokens need to be unequivocally recognized as the fundamental unit of both cost and value, making their consumption quantifiable and attributable. Second, the point of cost creation, specifically at the model call boundary, must be accurately tracked to pinpoint where resources are being utilized. Without proper attribution—knowing which agent or run initiated a particular model call—control becomes impossible. Consequently, the system should prioritize policy-based intervention over simply halting operations once a budget is exceeded, allowing for more nuanced and continuous management.

Current model gateways, while useful for certain functions, largely fall short in providing the granular control needed for AI agent cost management. Most existing frameworks primarily monitor at the model request level, offering capabilities such as model routing or rigid budget capping. However, what is critically needed is a system that monitors consumption at the individual run level, allowing for real-time visibility into the exact cost implications of each agent's execution. This run-level monitoring is essential for fine-grained control and effective policy application, moving beyond broad, reactive measures to proactive, precise management.

Boundary annotation serves a critical function by acting as a bidirectional communication channel, enabling the control plane to push actions directly down to the AI agent. This mechanism prevents a one-way communication stream, allowing for real-time adjustments to agent behavior based on predefined policies or dynamic conditions. Such direct interaction is essential for optimizing token usage and for ensuring that more agent runs can be successfully completed within their allocated budget caps, thereby improving efficiency and cost-effectiveness.

Consider an example where a Retrieval-Augmented Generation (RAG) tool unnecessarily generates 20 chunks of information. The control plane observes this behavior and, recognizing the inefficiency, intervenes to limit the output to a more appropriate five chunks. This action is pushed down via boundary annotation and then executed by a 'governor node,' which ensures that the change is applied in a non-destructive manner. This dynamic steering prevents excessive token consumption while allowing the agent run to continue, optimizing resource usage effectively.

The segment layer within the control plane is crucial for attributing agent runs and applying budget controls at a granular level. This allows for the creation of user cohorts, such as 'AIE 2026,' based on specific tags or dimensions, and then applying distinct budgets to these groups. This approach provides flexible rollup capabilities and enables fine-grained control over spending, ensuring that resources are allocated and managed according to the needs of different user segments or projects.

The FinOps solution distinguishes between 'halt' and 'steer' actions to manage AI agent behavior more effectively. While halt actions simply terminate an agent run when its budget is exceeded, steer actions represent a more advanced approach. With steer actions, the system intervenes to modify the agent's or its components' behavior, aiming to keep the run within its alerted budget without outright termination. This policy layer intelligently groups budgets and actions, allowing for more nuanced and continuous control over agent operations.

The implementation of control logic is facilitated through out-of-band boundary annotations, which means developers do not need to alter their core code. Instead, they simply apply annotations to the relevant methods. The control plane operates securely within the user's own tenant, and a governor instance requires configuration to define the specific actions it is permitted to take. This design minimizes disruption to existing codebases while providing robust, centralized control over AI agent behavior and spending.

The system's capabilities are demonstrated using a simple two-agent workflow. This pipeline consists of a research agent, which is equipped with access to a search tool, and a summarizer agent, responsible for compiling the final report. This basic setup effectively illustrates how policies and control mechanisms can be applied to manage the interactions and token consumption between different AI agents in a sequential task.

Preview mode offers a crucial feature for testing policies without immediate enforcement. In this mode, all defined policies run as they would in a live environment, but their actual enforcement mechanisms are disabled. This allows users to effectively calibrate thresholds and observe the potential impact of policies before they are fully implemented, ensuring that the guardrails are optimized and ready for live deployment without unintended consequences.

Integrating AI agent products into production environments necessitates a secure and systematic approach to governance. It is crucial to have a safe environment for initial deployment, allowing for thorough testing and tweaking of guardrails. This iterative adjustment process ensures that policies are optimized and effective in real-world scenarios before finalizing the cost thresholds. This careful fine-tuning guarantees that the production agents operate within defined financial and operational boundaries.

The FinOps solution clearly differentiates between 'halt' and 'steer' methodologies for AI agent control. 'Halt' behavior acts as a simple circuit breaker, immediately terminating an agent run if its budget is exceeded. In contrast, 'steer' behavior introduces predictive adjustments, working to keep runs active by dynamically modifying agent actions based on allotted budget and token consumption velocity. This steer approach aims to add significant value by preventing premature termination and optimizing the entire cost management scenario.

Dynamic steering is achieved by injecting specific guidance directly into the agent's system instructions. This guidance can be as straightforward as alerting the agent that it is running low on budget and instructing it to produce more succinct or summarized outputs. By subtly influencing the LLM's output behavior in this manner, the system effectively prevents premature run termination, allowing the agent to complete its task while adhering to budget constraints.

Benchmarking results for TokenOps, with the full policy suite enabled, demonstrate substantial efficiency gains. The average spend for AI agent workflows was reduced by almost 78%. Furthermore, compared to simple throttling methods that often kill agent runs, TokenOps significantly improved completion rates, which rose from 67% to approximately 96%. These benchmarks were conducted on open-source repositories such as browser-use and MetaGPT, providing empirical evidence of the solution's effectiveness in both cost reduction and operational reliability.

The current policy catalog has been meticulously developed through research into various prevalent failure modes in AI agents. It encompasses policies across several critical areas, including spend management, context management (e.g., context compaction and tool output reduction), loop detection, and progress detection. To address these diverse failure modes, the system supports a range of action types: Allow, Mutate, Inject, and Kill, providing a comprehensive toolkit for managing and optimizing AI agent behavior.