AlphaFold's Protein Puzzle: Unsolved Mysteries Remain

Rather than pioneering new biological discovery, Google DeepMind’s Pushmeet Kohli defined the AlphaFold process as a method to replicate structures already cataloged in the Protein Data Bank. The model's success in this replication, while useful, does not equate to a full understanding of protein dynamics.

The fundamental challenge lies in knowing the true ground state and the full distribution of protein structures, which AlphaFold has not yet captured.

The central query is whether simply scaling data, like scaling compute, will automatically lead to better outcomes, or if success depends on finding a specific scaling law where more input invariably yields superior results.

Sal Candido emphasized that scaling laws are not universally applicable, and identifying where they apply constitutes a significant part of research effort.

A model's capabilities are intrinsically limited by the statistical information contained within its training data, meaning effective scaling only occurs when the data provides the necessary insights to solve the target problem.

Researchers often lean towards using readily available data, even if it is not the highest quality or most relevant for pressing translational medicine challenges.

Sal Candido confessed to being 'lazy' and working with accessible datasets, citing the use of metagenomic sequences for training protein language models.

Despite its lower quality, training on metagenomic data can unexpectedly enhance the performance of models designed for real-world protein engineering, yet there is a risk in prioritizing easily generated data over problem-specific data.

Pushmeet Kohli noted that data generators often favor easily producible data, potentially limiting the scope and impact of research.

By fostering open, community-led initiatives, researchers can collaboratively identify and generate the precise data needed to discover effective scaling laws and drive scientific progress.

Pushmeet Kohli explained that the 'Bitter Lesson' should not be narrowly interpreted as only about scaling, but rather about avoiding rigid adherence to a single type of solution.

Researchers must prioritize the problem first, adopting a flexible approach that integrates modeling, data collection, and deep domain expertise.

He stated that defining oneself as solely a 'modeler' or 'data person' and applying a singular strategy, such as just optimizing a model on existing data, often leads to inadequate results.

AlphaFold's development relied heavily on existing, high-quality data from the PDB because generating additional data was not feasible due to a lack of expertise and resources.

In this context, DeepMind prioritized investing in modeling to maximize the impact of available data, contrasting with areas like cell genomics (Cell x Gene), where existing data proved insufficient for building a 'virtual cell.'

Researchers need to critically assess whether their primary constraint is data quality, data availability, or the capability of their modeling approaches to allocate resources effectively.

Early in their careers, researchers should define core objectives and evaluate constraints on data and compute to identify which scientific approaches are actually feasible.

This approach involves identifying and analyzing constraints early—such as limitations in computing, data generation, and model training—and being prepared to 'fail fast' to pinpoint viable long-term solutions.

The discussion centered on whether research resources should focus on developing artisanal, handcrafted solutions or move towards more generalized scaling approaches.

Pushmeet Kohli advocated for integrating scientific intuition from biophysics into AI models, arguing that this provides a 'data-efficient' advantage by reducing the need for models to replicate known scientific principles, making the curation of high-quality data as critical as the modeling itself.

Sal Candido noted that while inductive bias is crucial for models with limited data, it can become a hindrance if the underlying assumptions are incorrect as data volume increases.

He argued that scaling itself demands significant 'craft' and algorithmic work, including architectural evolution beyond standard transformers, to optimize performance on massive datasets.

Brandon Anderson highlighted that despite public perception that protein folding is 'solved,' researchers recognize substantial unresolved issues in protein biology.

Five years after the AlphaFold 2 announcement, fundamental problems such as protein function, dynamics, and design are still largely 'wide open.'

The core debate continues to be whether the main bottleneck lies in data availability, algorithmic limitations, or simply the passage of time needed for further advancements.

Pushmeet Kohli proposed a shift in AI training methodologies from relying solely on PDB structures to directly utilizing raw cryo-electron micrographs.

He explained that raw micrographs contain a richer dataset, including extensive distributional information and dynamic data, which deposited PDB structures often lack.

Directly modeling from these source images could overcome current limitations and offer a more comprehensive understanding of protein behavior.

Brandon Anderson posed the question of whether models should help humans understand underlying mechanisms or simply deliver functional outputs, referencing Richard Feynman's quote: 'what I cannot create, I do not understand.'

The discussion explored the need for scientific insight versus the utility of a 'magical black box' that can produce desired results, such as a picomolar binder.

Sal Candido acknowledged that effective black box design existed long before AI, but maintained that understanding remains crucial for scientific advancement and trust in results.

Sal Candido asserted that current protein language models contain more latent information about protein function and motion than is currently utilized.

DeepMind's research into interpretability aims to extract the vast evolutionary knowledge compressed within these models during their training.

These models effectively function as 'world models,' having condensed extensive evolutionary data into predictive structures that hold untapped insights.

Pushmeet Kohli emphasized that even a highly accurate model, such as AlphaFold at 95 GDT, would be unreliable without robust calibration of its uncertainty measures like pLDDT scores.

Such calibration is vital to prevent scientists from misdirecting efforts based on confident yet incorrect predictions.

Kohli distinguished between understanding how a model behaves and understanding the precise mechanism by which it arrives at its solutions, highlighting the former's importance for trust and utility.

Pushmeet Kohli questioned the definition of 'interpretability,' asking 'interpretable by whom?' when discussing complex AI models.

He argued that the inherent cognitive and computational limitations of the human brain make sophisticated models like AlphaFold 2 fundamentally uninterpretable to humans, suggesting that larger, more advanced AI systems might be better equipped to interpret other AI models.

According to Pushmeet Kohli, whether a human scientist or a future frontier model reviews the data determines the specific level of interpretability.

He suggested that if future large models were given access to the activation layers of models like AlphaFold 2, they might develop theories explaining how these results are produced, potentially offering a form of interpretability beyond human cognitive capabilities.

Sal Candido argued that focusing on achieving a 10x improvement rather than an incremental 10% gain encourages a broader perspective in problem-solving.

This approach allows researchers to identify novel solutions that might be overlooked when pursuing smaller, incremental advancements, driving innovation from fundamental principles.

He noted that Biohub’s mission to 'cure all disease' necessitates the pursuit of these transformative, 10x strategies, emphasizing that both incremental and radical research approaches should coexist.