AI Agents Become Distributed Systems

"it has turned into a distributed systems and it is important to incorporate distributed systems thinking and concepts when building AI agents."

Real-world incidents involving AI agents highlight the critical need for sound systems thinking in their development. Examples include a Replit AI instance deleting a production database due to unchecked authority and an Air Canada chatbot providing incorrect refunds because it lacked mechanisms to verify truth. These incidents underscore that many failures could have been prevented by applying robust distributed systems principles during the design phase.

"both of these incidents or a lot of these incidents could have been prevented by good systems thinking when building these AI agents."

The architectural boundary of AI systems has expanded significantly, moving beyond the confines of the Large Language Model (LLM) itself. AI agents are now capable of causing tangible side effects in the external world through their use of tools and changes to external states. This new capability requires designers to carefully consider how credentials, external states, and action boundaries are managed within these systems.

"the difference is that it can now cause side effects in the outside world."

Traditional software systems typically rely on deterministic, precisely mapped-out workflows, where outcomes are predictable. In contrast, an AI agent functions as a probabilistic coordinator, meaning the types and sequences of actions it can take vary significantly, leading to non-linear and less predictable behavior. Therefore, incorporating deterministic controls is crucial to constrain the potentially unpredictable actions of an AI agent and ensure reliable operations.

"In the case of AI agent, the AI acts as a probabilistic coordinator. The amount of action the kind of actions that it can take can vary quite a lot."

Each stage in this loop presents distinct systemic risks, including the potential for incorrect data persistence, the performance of undesirable actions, or the initiation of harmful retry storms. Consequently, every boundary within this agent loop requires rigorous management to mitigate these risks and ensure stable operation.

"It can persist incorrect data or and when deciding, it might also decide to perform an incorrect action, or worse, it It also do a retry storm."

By recording its state at each stage, the agent can recognize where an operation failed and perform reversible actions or compensation. This capability is essential for managing partial failures and maintaining overall system integrity.

Explicit transactions are also vital to maintain consistency during multi-step processes, especially when an agent performs operations that modify external states. When an agent experiences a timeout, it doesn't necessarily mean failure but rather an unknown state, requiring the use of request IDs, idempotency keys, and status queries to determine the actual outcome. Implementing strategies such as retry budgets, maximum turns, and exponential backoff is crucial to prevent retry storms that could overwhelm downstream systems and ensure that retries are managed efficiently without causing cascading failures.

Furthermore, handling memory in AI agents involves recognizing that any context influencing agent behavior is effectively part of its state. This necessitates treating memory as a cache that supports invalidation and clear provenance tracking. To enhance reliability, mechanisms such as circuit breakers, compensation actions, scoped credentials, and human approval workflows for specific actions are indispensable, transforming AI agents into more robust and accountable distributed systems.

"it is important to persist that so that if anything fails, the agent is able to recognize where it failed and it can perform a reversible action."

This measure ensures that the agent does not overwhelm or "burn out" these systems, especially if they are unhealthy or saturated.

"it is important to have uh circuit breakers whenever it is making external calls uh to ensure that the uh that it is not uh burning the downstream system."

Assigning rate limits and budgets to AI agents is essential to prevent them from incurring excessive costs, particularly through infinite retry loops or uncontrolled operations. Key constraints include setting a maximum number of turns, limiting parallel calls, and establishing a maximum spending budget. These measures are crucial for maintaining operational efficiency and financial control.

"It is also important to assign rate limits and budgets. Uh an agent can uh go over uh can run your cost uh if it's not assigned proper budgets and rate limits."

It is critical to move away from granting blanket permissions to AI agents, which can pose significant security risks. Instead, agents should be given scoped credentials, with separate read and write permissions to limit their access. Additionally, implementing an allow-list for the tools an agent can call minimizes the potential for harm, even from seemingly harmless models.

"it is important to give scoped credentials to it. There should be separate read and write permissions, and there should be allow list for the tools that it can call."

Approvals should be linked to the action itself, a timestamp, the actor, and an expiration. This precise linking prevents a small authorized action, such as a refund, from escalating into a larger, unauthorized one.

"a human approval shouldn't be tied uh to a blanket approval. It should be tied uh to uh action, timestamp, actor, and expiration."

Comprehensive observability is paramount, requiring teams to reconstruct the full context of an agent's failure: what happened, what information it was reacting to, and why it failed. This involves tracking not just prompts and responses but also tool calls, retrieved context, and approval statuses.

"observability is an important requirement uh when building AI agents because uh and logs are not enough. Teams need to reconstruct when an agent failed, what happened, what information was it reacting to, and why it failed."

Model intelligence alone cannot eliminate common distributed system issues like network failures, stale data, or adversarial inputs. Therefore, the focus should remain on bounding agent actions, observing their behavior, and establishing mechanisms for recovery.

"I would like to end with the idea that yes, model capability matters. Having good models improves the likelihood of it making correct operations... However, it cannot eliminate network failures, stale data, or adversarial input."

These contracts ensure that the agent is only allowed to perform operations within the scope of its provided permissions. Furthermore, all tools must incorporate idempotency, which guarantees that repeated requests will produce the same outcome and prevent dangerous side effects.

"It is important to have tool contracts in place to ensure that it is only allowed to make operations that it is given that it is provided the contract... and all these tools have item potency baked into it."