Preventing AI from Leaking Company Secrets

A primary concern in developing a company brain, an internal AI system, is the risk of leaking sensitive information. Tanmai Gopal highlights that a significant fear is unauthorized access, such as an intern potentially viewing compensation details. This security vulnerability currently represents the most substantial obstacle to the widespread adoption of enterprise AI systems like PromptQL or Hermes.

PromptQL's team brings significant expertise in managing complex data access challenges, stemming from their work on the Hasura GraphQL engine, a popular open-source project. This background involved solving intricate data access problems for major corporations such as Apple, Meta, and JP Morgan. Their experience with these high-profile clients has shaped their approach to securing enterprise AI systems.

Companies adopting AI systems fall into three distinct categories, each with unique security demands. AI-native companies prioritize functionality, willing to implement solutions as long as they work effectively. Tech-forward companies, exemplified by Instacart, seek best-of-breed technology and operate with agility. Fortune-tier banks, however, impose rigorous security regulations to safeguard highly sensitive financial data.

PromptQL models its internal company brain as a wiki, comprising approximately 5,000 interconnected pages. Company brains can be structured in various ways, including markdown files stored on GitHub or complex knowledge graphs. This wiki-based approach allows for flexible organization and retrieval of information.

The consistency of an AI company brain's update frequency serves as a key indicator of its health and ongoing utility. If one were to plot the daily number of updates, different trends would emerge, each signaling a distinct state of the system's engagement. A declining trend typically suggests a loss of initial enthusiasm among users after the system's deployment, indicating it might not be integrated effectively into daily workflows. Conversely, a fluctuating trend points to varying levels of individual user engagement, where activity spikes and drops depending on specific tasks or user interests. Most favorably, a steady upward trend signifies a robust and effectively self-learning system that is continuously growing and evolving, demonstrating sustained user adoption and increasing value over time.

A healthy company brain exhibits specific growth patterns in its daily updates, reflecting its increasing utility and user engagement. As users teach the AI agent more skills, the number of daily updates tends to steadily increase over time. Initially, agents might be tasked with fundamental operations such as querying data. However, as the system matures, users begin to assign more complex and subsequent tasks, like interpreting data or conducting A/B testing, which further drives the volume of daily updates. This continuous expansion of capabilities results in a healthy brain displaying a gentle and consistent upward curve in its system activity, indicating sustained learning and effective integration.

Company brains are primarily deployed in enterprise environments for two main categories of use cases. One category involves individual utility for specialized tasks, such as swiftly answering complex security questionnaires, where the AI provides rapid access to specific, organized knowledge. The second, broader category is shared collaborative usage, often facilitated by AI agents integrated into communication tools like Slack, allowing teams to leverage collective intelligence. A significant application within this collaborative sphere is incident management, where the AI system can streamline workflows and provide crucial information during team-wide crises, enhancing overall operational efficiency.

The company brain architecture is defined as shared context, typically stored in a collection of markdown files. This shared context also includes access control rules that govern how coding agents can interact with various data sources and tools. This specific architectural design is explicitly tailored for coding agents, rather than being a general-purpose AI model, ensuring controlled and secure operations.

PromptQL's strategy for building company knowledge systems explicitly rejects the traditional approach of constructing gigantic, centralized knowledge graphs for an entire organization, arguing that such systems have historically failed and will continue to do so. Instead, their system focuses on developing agents capable of solving general-purpose problems through code, which can adapt and evolve. The core proposal is for a decentralized model where each individual within the company is responsible for owning and building their specific, relevant part of the company brain. This approach contrasts with the unfeasible task of creating a single, massive, enterprise-wide knowledge graph and instead fosters organic, distributed growth of organizational intelligence.

A critical security measure for managing internal company knowledge involves implementing granular file-level access controls. This means defining explicit read and write permissions for every file within the company's wiki. Such a system ensures that users have precise control over who can access specific pieces of information, preventing unauthorized viewing or modification of sensitive data.

To maintain accuracy and accountability within the company brain, it is crucial to disable the automatic addition of AI memory. This rule prevents agents from autonomously adding information to the system without human oversight. By requiring a human-in-the-loop, organizations retain full visibility into what information is being recorded, ensuring accuracy, preventing the spread of incorrect data, and maintaining accountability for all knowledge contributions.

The recommended approach for integrating AI into knowledge management is to have agents suggest content, along with appropriate security scopes, for human approval. This method provides a crucial middle ground between entirely manual pull request (PR) reviews and completely uncontrolled automation. By allowing the agent to suggest what is added and with what permissions, human decision-makers can efficiently accept or reject proposed changes, ensuring accuracy and security while leveraging AI for content generation.

A streamlined workflow for validating knowledge updates involves AI agents suggesting content, which users then review and approve. For instance, an AI might analyze an email thread and suggest a list of bullet points to be added to the wiki. Users can then quickly review these facts for accuracy, focusing on the content rather than technical formatting, and assign specific security scopes to the new information before it is formally added. This process significantly simplifies and accelerates the contribution of knowledge while maintaining human oversight.

A fundamental rule for maintaining a company-wide wiki is that every entry and change must be explicitly tied to a specific human owner. This means that content additions cannot be attributed generically to an AI agent like 'Claude' or 'Hermes'; instead, they must be recorded as, for example, 'Tanme added this.' This mandatory human attribution ensures clear accountability for all information within the wiki. In the event of a sensitive data leak, this system allows for immediate remedial action and clear identification of responsibility, reinforcing the integrity and security of the knowledge base.

The security architecture for agents dictates that an agent must use the specific user's claims when reading context to perform a task. This means if a user is solving a finance problem, the agent will utilize that user's finance claim to access information as if it were the user themselves. This mechanism ensures that the agent only accesses information that the initiating user is already authorized to view, thereby maintaining strict security protocols and preventing unauthorized data exposure.

AI systems can significantly enhance troubleshooting by critically evaluating inefficient queries and offering improvements through iterative learning. For instance, an AI might surface knowledge by identifying that an 'equals to' query is more efficient than a 'like' query for certain operations. The system can pinpoint specific lines of code causing errors and, through these interactions, the agent learns and subsequently offers persistent knowledge updates, improving its own performance and the overall efficiency of the system. This process demonstrates the AI's ability to self-critique and evolve its problem-solving capabilities.

Autonomous agents capable of performing multiple tasks pose significant security risks, particularly concerning privilege escalation. It is dangerous if an engineer can use the same AI agent for debugging tasks and then, without proper checks, for deploying code to production environments. Such an integrated capability can lead to serious security breaches, especially in highly regulated sectors like banking, which demand a strict separation of concerns. To mitigate these risks, the speaker proposes injecting user credentials at the HTTP/SQL layer, ensuring that even within an autonomous workflow, access remains aligned with the user's specific, limited permissions, thereby maintaining safety and accountability.

The core architectural principles for ensuring AI safety, particularly in managing credentials and context, can be summarized into two rules. Firstly, user credentials should never be stored within the cloud sandbox where the AI operates. Secondly, all interactions with real data must be virtualized through proxy layers, rather than directly. These rules ensure that users maintain control over access to specific tools based on their individual credentials, significantly enhancing security by isolating sensitive information and operations.