Jeff Dean Launches Discovery Loop to Automate Scientific Discovery, Emphasizing AI's Societal Impact

Jeff Dean has co-founded a new company, Discovery Loop, established as a public benefit corporation. Its ambitious mission is to automate machine learning science and engineering to accelerate the rate of discoveries across numerous scientific disciplines. Dean stated, "the idea behind our new company Discovery Loop which is a public benefit corporation and our you know fairly ambitious mission is we want to automate machine learning science and engineering so that we can improve the rate of discoveries across many different fields."

The company is specifically targeting a significant reduction in the time it takes to conduct experimental iterations, aiming to compress cycles that currently take days into mere minutes. This approach involves creating multi-agent systems designed to emulate the expertise and workflow of PhD-level researchers in various domains.

A notable contrast was highlighted between Jeff Dean's decision to start a new company and Dawn Song's move into a large platform. The moderator pointed out, "Don is moving into large platform and Jeff is leaving a very large platform to start something new. Apparently even among the two best people today on stage cannot agree what is the optimal size of a company doing AI." This illustrates an ongoing debate within the AI community regarding the most effective organizational scale for advancing artificial intelligence. Some prominent figures opt for the vast resources and established infrastructure of large corporations, while others believe that the agility and focused vision of new ventures are better suited for groundbreaking innovation.

The moderator emphasized a crucial shift in AI's developmental trajectory, moving beyond mere capability enhancements to address real-world societal impact. "The last decade has been about making AI more capable. Now the next chapter raises the bar for us to have to answer even more important question. What can we actually accomplish with all of this great intelligence? Can AI cure cancer? Can AI make our society better?" The previous decade largely concentrated on refining AI's core functionalities and expanding its processing power. Now, the conversation is pivoting towards practical applications, posing critical questions about how AI can actively solve complex global challenges like curing diseases or improving societal well-being.

Jeff Dean delved into the foundational idea behind his influential 2017 research, 'Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.' The primary objective of this work was to develop extremely large models possessing vast capacities to store and process information. Dean explained, "the intuition behind thee work was really that you want to have really really large models that have a lot of capacity to remember lots of things, but that you want to make them efficient by only calling on the parts of the model that are most useful." This modular architecture allows the model to selectively activate only the most relevant 'expert' components for a given task, thereby maintaining efficiency despite its massive scale.

The development of sparse activation models, particularly the Mixture of Experts, led to a significant leap in computational efficiency for AI training. Unlike dense models that activate all parameters for every input, sparse models engage only specific 'expert' components relevant to a given task, analogous to how the human brain doesn't utilize all its regions simultaneously. Jeff Dean expressed excitement over this breakthrough, noting, "So we were pretty excited about that. We that was one where we sort of knew it was going to be a big deal because we could see like 10x better sort of uh training compute to quality ratios than if you had dense models which were kind of the main thing people were using at that time."

This innovation yielded a tenfold improvement in the training compute-to-quality ratio, marking a substantial advance over the dense models predominantly used at the time. This efficiency gain allowed for the creation and training of much larger and more capable models without a proportional increase in computational resources, paving the way for future large-scale AI architectures.

Around 2011-2012, Google initiated a strategy of significantly scaling up deep learning models, leading to dramatic reductions in error rates across various applications. Jeff Dean cited, "you would get like amazing reductions in you know uh image classification error rates uh speech recognition improvements were were kind of like equivalent to the previous 20 years of speech research in terms of lowering error rate by just scaling up a fairly simple deep acoustic speech model." By creating neural networks 50 times larger than previous iterations, substantial improvements were observed. For instance, the advancements in speech recognition performance achieved in a short period through model scaling eclipsed the cumulative progress of two decades of prior research.

TensorFlow was developed with a core objective of abstracting the complexities of hardware implementation from machine learning researchers and developers. This elegant abstraction allowed users to focus on model design rather than the intricacies of how their models executed on computational hardware. Dean explained, "we wanted to bring that kind of uh you know the ability for the ML researcher or developer to be a little bit abstracted from exactly how it's mapped onto the compute hardware because that's a really nice elegant abstraction. you can say I just have this beautiful ML model in my head and I don't care how you implement it but please make it run really fast."

By enabling users to express their models through computational graphs, TensorFlow provided a powerful interface. Open-sourcing the platform further standardized research implementations, fostering a collaborative environment where developers could efficiently build and deploy complex machine learning models without deep hardware expertise.

The Gemini project was co-founded by Jeff Dean and Oriol Vinyals with the explicit goal of consolidating disparate AI research efforts. Dean recalled writing a one-page memo, stating, "I wrote a onepage memo. I'm like this is just silly. We should just all work together. Let's combine our uh people and ideas and uh compute resources and train one you know model that is multimodal from the start." The initiative brought together the legacy teams and resources of Google Brain and DeepMind, aiming to streamline development and focus on a single, powerful AI model. This strategy was centered on developing a truly multimodal architecture from its inception, capable of processing and understanding various types of data comprehensively.

Jeff Dean acknowledged an initial oversight in Gemini's development, specifically regarding its coding capabilities. He admitted, "I think maybe our focus on making it amazing at coding was lagging a little bit and we realized that and are you know work to catch catch up on that." Initially, the focus might have been too broadly distributed, leading to this gap. Recognizing that strong coding capabilities are intrinsically linked to better reasoning skills in AI models, corrective efforts were promptly implemented to enhance Gemini's proficiency in coding tasks. This strategic pivot aimed to improve the model's overall intelligence and problem-solving abilities.

Jeff Dean shared his heuristic for fostering research discovery, emphasizing a broad, rather than deep, initial approach to literature review. He often advises students, "I often tell students it's better to to skim 10 papers than to read one in detail because you kind of then get 10 points in your cloud of like what might be possible." The goal is not to master every detail of a single paper, but to rapidly absorb a multitude of diverse ideas.

This method aims to build a conceptual 'cloud' of possibilities, enabling researchers to connect previously unrelated concepts. Dean believes that this interdisciplinary approach to 'connecting the dots' is crucial for structural problem-solving and generating truly novel insights, rather than incremental improvements within a narrow field.

The application of AI in cybersecurity presents a dual-use dilemma, offering both powerful defensive capabilities and significant offensive potential. Jeff Dean explained, "They can also be used to find security vulnerabilities, and that's a double-edged sword. You could use them to patch security vulnerabilities that exist and there are lots of them in the world and you could also as if you're a malicious user you could use them to to exploit them." AI tools can efficiently identify weaknesses in systems, which can be leveraged by ethical practitioners to patch existing vulnerabilities.

However, the same advanced tools can be wielded by malicious actors to exploit these very flaws, creating a more sophisticated and rapidly evolving arms race between cybersecurity defenders and attackers. This inherent duality underscores the critical need for responsible development and deployment of AI technologies in the security domain.

Jeff Dean detailed the concept of recursive self-improvement in AI, where models are designed to generate and optimize their own architectures. He noted, "My colleague and actually co-founder Quoc Le and others sort of kicked off some of the early work in this space with neural architecture search which is a way in which you can have a model generating model that generates machine learning model architectures and then evaluates how well those model architectures work." This process, known as Neural Architecture Search (NAS), involves an AI model creating different machine learning model architectures. A reinforcement learning mechanism then provides feedback, evaluating the performance of these generated architectures and guiding the model towards more optimal designs.

This iterative process allows for continuous refinement and optimization. A significant example is the Evolved Transformer, which demonstrated a 30% increase in efficiency over the standard Transformer model, illustrating the profound impact of models designing their own improvements.

Discovery Loop, established as a public benefit corporation, is founded on the ambitious mission to automate machine learning science and engineering. Jeff Dean articulated, "The idea behind our new company Discovery Loop which is a public benefit corporation and our fairly ambitious mission is we want to automate machine learning science and engineering so that we can improve the rate of discoveries across many different fields." The company's core strategy involves deploying sophisticated multi-agent systems designed to mimic and surpass the expertise typically found at the PhD level across a diverse array of scientific domains.

This approach is aimed at drastically reducing the time required for experimental iterations. By automating complex research workflows, Discovery Loop seeks to shrink discovery cycles from days to mere minutes, thereby accelerating the pace of scientific breakthroughs and innovation on a grand scale.

Discovery Loop's identity as a public benefit corporation underscores its commitment to prioritizing societal well-being over purely financial interests. Jeff Dean clarified, "as a public benefit corporation we want to bring those scientific discoveries to as many people in the world as possible. So our goal will be to distribute those widely and we might make decisions that are not in the company's financial interest but are in the broader societal good." This designation means the company is legally obligated to consider the impact of its decisions on society and the environment.

Therefore, the primary objective is to disseminate scientific discoveries as widely as possible, ensuring that breakthroughs benefit the greatest number of people globally. This might entail making strategic choices that do not necessarily maximize immediate corporate profit but instead serve the broader public good.

Dawn Song elaborated on her decision to integrate her startup with Meta, citing the platform's extensive reach and established infrastructure as key factors. She noted, "Meta I think also has a great platform. I think it's really meta Google like really aggra the the biggest distribution platform for meta with the billions of users and hundreds of millions of um businesses." Song's evaluation was based on Meta's capacity to provide a massive distribution network, leveraging its billions of users and hundreds of millions of businesses.

Having previously exited other startups, Song's choice reflects a strategic alignment where her company's technology could scale rapidly by combining with Meta's immense user base, offering a powerful avenue for impact and further development.

Jeff Dean explained the factors enabling startups to successfully compete with large technology firms in the current AI landscape. He highlighted, "there's the rise of cloud computing and you know deployments of very large amounts of ML oriented compute in various cloud platforms and so that makes it possible for you know a small set of people to you know raise a fair bit of capital and be able to use those platforms without having to build them themselves." The advent of robust cloud computing services has significantly lowered entry barriers, allowing smaller teams to access massive amounts of machine learning-oriented computational power without the need to build and maintain their own infrastructure.

Furthermore, startups can avoid the internal distractions and bureaucratic overhead often inherent in large corporations. This agility enables them to maintain a sharp focus on specialized engineering goals, such as scientific automation, thereby achieving breakthroughs more efficiently than their larger counterparts.