Why Enterprise Tech Stacks Aren't Ready for AI Agents: Tackling Security, Compliance, and Audit Trails

Christopher Lovejoy, a technical staff member at Anthropic, works as a deployed engineer, integrating within enterprise organizations to help them leverage AI agents. Previously, he collaborated with Saul Howard, the VP of Engineering at Anterior, a New York-based company that supplies AI to US health insurance firms. Both have extensive experience building and deploying AI solutions in enterprise, especially within the highly challenging and regulated healthcare sector.

The speakers emphasized that lessons learned from healthcare enterprises, due to their significant impact on people's lives and strict regulatory requirements, are transferable to other regulated industries. This includes finance, defense, and government work, where process adherence is critical. This presentation aims to share insights on why current enterprise tech stacks are ill-equipped for AI agents and to introduce architectural principles to address these shortcomings, using a healthcare workflow as a primary use case.

A typical AI project in an enterprise begins by identifying a customer and a priority use case, such as an administrative healthcare workflow. This involves scoping a Proof-of-Concept (POC), defining key metrics for benchmarking, and allocating engineers for a build-out that usually spans about four weeks. The enterprise tech stack, though complex, generally involves an application layer, a control plane, and a data plane, all of which the POC needs to access. The POC solution connects to various data sources, including offline data pulling and online access, demonstrating its functionality.

After a successful POC, stakeholders typically react positively, with finance chiefs eager to understand budget impacts, medical officers highlighting AI accuracy, and sales heads wanting to promote "powered by AI" features. However, this initial enthusiasm often overlooks the significant challenges of moving the AI into production. The assumption that the hardest part—the AI itself—is complete is a common pitfall. The real difficulties emerge when stakeholders inquire about audit trails for compliance or the handling of sensitive data during discussions about productionizing the application.

As conversations shift to production, critical questions arise regarding audit trails and data integrations, often proving challenging for existing systems. Stakeholders also raise concerns about sensitive data handling, questioning how it's passed to agents, whether strict data boundaries are respected, and how the system ensures data privacy. The chief medical officer often inquires about the approval process for AI decisions, especially in scenarios requiring clinician escalation. Additionally, questions emerge about model manipulation by untrusted data, ongoing performance validation, and integration with established enterprise systems like Epic and Salesforce. The presentation focuses on the audit trail, data handling, and escalation issues.

Security teams demand an audit trail, which for programmers often sounds like a standard developer log found in tools like Data Dog. However, in enterprise security frameworks such as Sock 2, High Trust, and HIPAA, an audit trail is far more comprehensive. It must be a complete, immutable record of every event in the system. Legally, if an agent's decisions were challenged in court, the audit trail must provide a justifiable chain of evidence for every action taken, demonstrating transparency and accountability.

When architecting such systems, the goal is to make certain aspects easy by carefully choosing constraints. A key architectural principle is an immutable, append-only timestamp log. This log is designed to be complete and unified, serving as the single source of truth for all system data across parallel-running agents. This design choice, while a tradeoff, simplifies the process of tracking and verifying system state over time.

This event logging paradigm ensures that it's always possible to roll back time and reconstruct the exact state of the system at any given moment, providing a reliable audit trail. The primary trade-off is that while writing events is very easy, merely dropping an event, reading and reconstructing a view of past events can be more complex. However, patterns like caching and snapshots can be implemented to mitigate this challenge.

The event logging pattern simplifies creating different views of the source of truth at specific times, as all data views are ephemeral computed projections of the event log. The next critical challenge arises from compliance officers, who demand clarity on how sensitive data is handled and its lifecycle within the system. In healthcare, this concern is amplified due to the highly sensitive nature of patient information.

Healthcare data is notoriously complex, often unstructured or semi-structured, and does not always follow strict hierarchical relationships. Individual pieces of healthcare data can be very large, exceeding a megabyte, and require stringent access controls, including role-based access control (RBAC). Some customers may even require that their healthcare data never leaves their on-premise VPC. Schema-driven object storage is well-suited for these characteristics, aligning effectively with event logging.

To ensure immutability and robust observability, event logs, which record agent actions, only contain references to the schema-driven blobs where the actual healthcare data is stored. This separation means that developers cannot directly access sensitive data. By storing healthcare data immutably, it's always possible to reconstruct precisely what data an agent accessed at any point in time, enabling detailed debugging and traceability of agent steps and decisions. This design ensures that observability, orchestration, and instrumentation are decoupled from the sensitive healthcare data itself.

The separation of observability and orchestration from healthcare data also enables a zero-trust model, where object storage becomes a key enforcement point. Agents can use temporary tokens to access data only at the point of use, preventing data from flowing unnecessarily across the system. This design addresses the "lethal trifecta"—the risk of an agent accessing data it shouldn't, even if given legitimate access to other data within the same process. Orchestration logic provides a mechanism to solve this constraint.

A critical challenge in AI agent systems is dynamic escalation, where an agent's decision or action needs to be routed to a human for approval. This can be triggered by predefined rules, such as a medical treatment exceeding a certain threshold. The difficulty lies in seamlessly transferring context, as Large Language Models (LLMs) can handle vast amounts of text, but humans cannot. This disparity makes it challenging to maintain a coherent context across agent and human interactions.

A powerful pattern for dynamic escalation is to define "agent" more broadly, encompassing both LLMs and humans within the platform. This approach ensures that any action an LLM can take can also be performed by a human. This enables seamless escalation at any point in the agent's action chain, as downstream steps are indifferent to whether an LLM or a human performed upstream actions. This wider definition also facilitates context sharing by establishing shared definitions of context, independent of the actor.

Evaluating AI models (evals) is crucial for driving decisions about model selection and product approach, but it presents significant challenges. LLMs are non-deterministic, making consistent evaluation difficult. Additionally, offline datasets can quickly become outdated. The three architectural primitives discussed—the immutable ledger, orchestration-adjacent object storage, and human-agent equivalency—offer a pathway to effective, privacy-preserving evaluations, even in production environments.

The immutable ledger allows for action replay, enabling developers to revisit any point in the sequence of events, reconstruct the complete system state, and make specific tweaks to prompts, models, or code. This provides a direct impact assessment of changes. Furthermore, orchestration-adjacent object storage facilitates running evals on production data, even within customer environments, without exposing sensitive information. These capabilities enable robust and privacy-preserving evaluations.

The four architectural principles crucial for building enterprise AI in regulated environments are the immutable ledger of actions, orchestration-adjacent object storage, and human-agent equivalency, along with the natural emergence of evals as a first-class property. This approach emphasizes taking architectural constraints seriously and designing for simplicity. These patterns, though existing in other enterprise contexts, need to be combined in innovative ways for AI. It's about thinking beyond traditional software models, as AI introduces unique challenges that require fresh architectural thinking.

The critical takeaway is that problems arise when enterprises try to build a production system by merely adding enterprise requirements onto an initial, promising proof-of-concept. This reactive approach often leads to brittle, hard-to-generalize solutions. Instead, success comes from embracing the constraints of a production system from the outset. By designing with these architectural principles in mind, companies can build robust, scalable, and compliant AI agent systems that meet the rigorous demands of regulated industries.