Adyen's Dhruv Ghulati on Building Trustworthy AI: From Governance to Open Ecosystems

Regulators are increasingly questioning the traditional "human-in-the-loop" paradigm, prompting organizations to reevaluate why human intervention was initially deemed necessary for processes like loan or mortgage applications. The key question now is whether human expertise is genuinely indispensable, or if a machine can perform these tasks with greater fairness and efficiency. Dhruv Ghulati emphasized that manual processes were established for specific, valid reasons that must be thoroughly understood before automating them.

Established companies typically implement highly restrictive data access models, characterized by locked-down user groups and stringent permissions where specific parts of the codebase are inaccessible to many employees. These common restrictions include disabled downloads, limited access to production databases, and ring-fenced tool access to maintain security. In contrast, early-stage companies often prioritize speed and innovation by providing universal data access to nearly all employees, creating a stark difference in their operational approaches.

Companies must critically examine whether their existing access controls are genuinely necessary by weighing the potential risks of granting access against the benefits of increased velocity. This involves a careful assessment, distinguishing between less severe risks like data misinterpretation and catastrophic possibilities such as malicious code deletion. Manual security permissions often create significant bureaucratic friction and internal bottlenecks, hindering efficiency and innovation within organizations.

AI agents present a significant opportunity to automatically generate intricate permission layers, thereby shifting the heavy burden away from manual interventions by security experts. This automation could revolutionize security management, which has traditionally been reactive, with issues only discovered and addressed after breaches occur. By proactively managing permissions, AI agents can enhance both efficiency and security posture.

AI-native companies are increasingly adopting Model Context Protocol (MCP) strategies to effectively manage and scale tool access for their AI agents. This approach provides a significant efficiency boost, akin to a "magical unlock," compared to traditional point-and-click interfaces. For example, an internal system leveraging MCP could allow Product Managers to edit numerous tickets in bulk via terminals, vastly improving workflow and productivity.

The current landscape of AI development mirrors the early transition of the API ecosystem, where the focus was on promoting open integration and interoperability. The true value now lies in creating comprehensive toolkits, developing essential skills, and identifying practical use cases for effective data interaction within an AI-driven environment. Companies should strategically evaluate whether their efforts are actively fostering an open ecosystem for AI agents, enabling broader integration and innovation. This approach is crucial for unlocking the full potential of AI and driving collaborative advancements across various industries, emphasizing shared resources and standardized interfaces for agents to interact seamlessly with diverse data sources and applications.

Defining what constitutes a 'good' versus 'bad' response in a conversation is highly subjective, making it challenging for AI systems. The primary difficulty lies in translating the nuanced and often qualitative aspects of human conversation into measurable quantitative metrics. This challenge requires sophisticated methods to capture the subtleties of interaction and evaluate performance effectively.

Reinforcement Learning from Human Feedback (RLHF) is a critical technique used to fine-tune the behavioral characteristics of language models. This process involves product work focused on understanding and defining the desired personality a model should exhibit. Companies like OpenAI and Anthropic strategically adjust their models to be more or less friendly, which is a key part of their product development and user experience alignment.

A significant challenge in AI product development is bridging the gap between internal model metrics and actual user growth or business outcomes. Improvements in internal metrics, such as a model's ability to adopt a friendlier tone, do not automatically translate into guaranteed business growth. User growth may instead depend on specific user needs, such as the accuracy of tool-calling capabilities, rather than superficial conversational style.

Most AI product development requires a clear separation of core ML metrics from broader online alignment metrics that reflect business outcomes. This distinction is crucial because failing to connect ML performance directly to product growth is a common pitfall in strategic planning. By identifying one or two core ML metrics that truly indicate a good ML product, organizations can better understand its impact on overall business objectives.

For enterprise AI implementation, foundational governance elements include robust access controls, comprehensive data catalogs, and thorough logging mechanisms. Organizations must meticulously track what data is being utilized by their AI systems and what new metadata sets are subsequently created. This rigorous oversight ensures data integrity, security, and compliance across the organization.

Historical datasets, such as previous loan applications, often contain societal biases related to sensitive features like race or geographic location. When AI models learn from these features, they can inadvertently perpetuate discriminatory decision-making. It is crucial to identify and mitigate such biases to ensure fair and equitable outcomes, especially in critical applications.

Implementing safeguards, particularly through red teaming, is crucial to verify if AI models are overly reliant on sensitive features that could lead to biased outcomes. Red teaming involves rigorously testing decision boundaries to ensure models do not perpetuate existing systemic flaws. This proactive approach helps in preventing unintended discrimination and building more equitable AI systems.

A common misconception is blaming the AI model when the root cause of an issue lies within the broader system-level behavior. While removing specific features might degrade a model's performance, it can significantly clarify the underlying decision-making processes. Understanding this distinction is vital for accurate problem diagnosis and effective system improvement.

By meticulously examining governance frameworks and understanding the factors that truly drive an ML model's decisions, organizations can significantly improve their overall systems. Insights gained from governance can initiate a complete re-architecture of operational processes, such as loan approval or fraud detection. This not only automates the system but fundamentally changes how these critical business functions are performed.

Companies face a critical decision: whether to prioritize building systems with the highest precision and recall, or to sacrifice some performance for greater explainability. Highly predictive features often lack logical interpretability, complicating the ability to understand why a model makes certain decisions. While maximizing raw metrics is tempting, interpretability is a valuable asset that contributes to long-term system improvement and trust.

CTOs cannot foresee every potential use case for machine learning within an organization, given that thousands of employees have tasks that could benefit from automation. Instead of attempting to meticulously map out all AI strategies in documents, it is more effective to invest in creating sandbox environments. These environments should provide access to anonymized business data and offer simple model templates, such as XGBoost, logistic regression, and Large Language Models, to encourage widespread experimentation. This approach empowers employees to discover and implement AI solutions organically, fostering a culture of innovation from the ground up.

The current era is characterized as the age of building and creation, presenting a significant opportunity for both workplaces and humanity as a whole. This period encourages individuals to actively engage with technology, being creative and hands-on in integrating AI tools into their daily work processes. Embracing this mindset allows for widespread innovation and personal growth alongside technological advancement.