Why Overconfident AI Needs 'Uncertainty' Math

Zoubin Ghahramani, a Cambridge University professor and Google DeepMind Research VP, asserts that intelligence is fundamentally tied to decision-making, which must occur within an uncertain reality. He emphasizes that both human and artificial intelligence operate with limited perception, making it impossible to know everything about the real world. Therefore, intelligent systems must be capable of representing and updating their understanding of uncertainty to make effective decisions.

AI models, despite being trained on vast datasets, can fail when confronted with rare or 'long-tail' scenarios not adequately represented in their training data. Ghahramani illustrates this with the example of a self-driving car encountering an unfamiliar hailstorm or an unexpected obstacle. An AI system equipped with uncertainty awareness would recognize the unusual situation and choose to slow down or request human intervention, rather than acting with false confidence, which could lead to dangerous outcomes.

Ghahramani differentiates between two crucial types of uncertainty: aleatoric uncertainty, which refers to inherent randomness in the world, and epistemic uncertainty, which stems from a lack of knowledge. He notes that understanding this distinction is vital because different types of uncertainty require different strategic responses. While random events may simply require accepting limitations, knowledge gaps necessitate actively seeking more information, formally defined as the reduction of uncertainty, to make more informed decisions.

Ghahramani points out that humans are surprisingly poor at explicitly estimating probabilities, often making errors by orders of magnitude when asked for conscious assessments. However, he explains that our evolutionary development has led to an unconscious mastery of probabilistic inference, which is critical for survival. This allows humans to instinctively process sensory information and integrate prior beliefs to navigate complex environments without needing explicit calculations.

Ghahramani provides a striking example: altering just a few pixels in an image of a school bus can cause a neural network to confidently identify it as a cheetah. This vulnerability underscores the need for AI systems that are not only accurate but also appropriately cautious and aware of their limitations.

Ghahramani observes that current large language models (LLMs) consistently deliver answers with absolute confidence, irrespective of their factual accuracy. He notes that if a user questions an LLM's response, the system frequently 'flip-flops' and provides a different, often contradictory, answer. This behavior highlights a critical flaw in their design, indicating a lack of true internal understanding or a robust mechanism for expressing doubt.

He reiterates this point, noting that while current LLMs utilize probability for tasks like next-token prediction, they often lack explicit, formal representations of uncertainty within their core architecture. This absence prevents them from accurately assessing and conveying the reliability of their own outputs.

Without a solid mechanism to represent probabilities over their own beliefs, AI systems struggle to exhibit genuine uncertainty or consistent internal coherence.

Ghahramani explains that AI systems struggle with uncertainty because their current training paradigms have not prioritized this aspect. Instead, models are trained to excel at 'modeling the data soup,' focusing on pattern recognition rather than developing a deep understanding of cause-and-effect relationships. He stresses that for safe operation in the real world, AI must be capable of understanding its own uncertainty and the causal mechanisms at play, which is currently a significant gap in their development.

Ghahramani reveals that the foundational ingredients for building rational AI systems existed 15 to 20 years ago. However, he explains that the vision was abandoned because representing full probability distributions over every possible factor was deemed computationally intractable at the time. This computational constraint led researchers to shift towards data-driven training paradigms. Ghahramani suggests that with contemporary advancements in computing power, revisiting these probabilistic approaches is now a viable and necessary path forward for AI development.

Ghahramani explains that the protein prediction is color-coded, with different hues indicating how sure the model is about the correctness of the folding. This visualization addresses the inherent physical 'wiggling' of proteins and helps distinguish between predictions based on experimental data and those derived solely from the model, using uncertainty clouds to illustrate molecular positions.

Ghahramani defines true calibration in AI as a state where the system's stated probability for an event precisely matches its real-world outcome frequency. He offers an example: if an AI predicts a 70% chance of rain, then across all instances where it made that prediction, rain should occur 70% of the time. While this definition is straightforward for repeatable events, Bayesian statistics extends this to non-repeatable events by allowing probability to represent subjective uncertainty. This precision is critical to prevent AI from merely hedging or failing to provide useful, actionable insights.

He notes that while current LLMs generally perform well, they tend to uncover significant gaps when confronted with 'long-tail' or unusual scenarios. He emphasizes that merely scaling compute and data is insufficient for high-stakes applications such as reliable self-driving cars or medical AI systems that aid in diagnosis. For these applications, he stresses, accurately quantifying probabilities and ensuring high reliability are paramount.

He explains that current AI models suffer from 'catastrophic forgetting,' where updating them with new data causes them to lose previously acquired knowledge. In contrast, human intelligence relies on constantly receiving streams of data and continuously adapting neural connections. Ghahramani points out that Bayesian updating offers a robust theoretical framework for enabling this type of continual learning, allowing AI to adapt without forgetting.

He notes that modern AI systems currently rely on approximations rather than exact Bayesian inference due to these computational demands. However, he optimistically states that advances in computing power and decades of research into efficient approximation methods make revisiting and implementing these theoretically ideal Bayesian ideas more viable than ever before.

He argues that for problems of significant importance, an AI system that knows its limitations and when it 'doesn't know' is far more valuable than one that is overconfident and frequently incorrect. He emphasizes the need for human-centric AI that transcends human flaws in reasoning, focusing on reliability and doubt as critical features for addressing societal needs effectively.