The Secret Race to Build Artificial Superintelligence and Its Moral Implications

Artificial Intelligence (AI) broadly encompasses machine learning systems and neural networks capable of absorbing data, learning, and taking actions. This general category includes all the AI technologies currently observed. Beyond this, there are more specific distinctions.

Artificial General Intelligence (AGI) represents the ambition of an AI system that can perform every cognitive task the human brain can, signifying human-level general intelligence. Artificial Superintelligence (ASI), in turn, refers to an AI that is vastly superior to all humans in every task. While AGI has been the primary focus of AI research for the past two decades, with leading companies like OpenAI, Anthropic, and Google DeepMind intensely pursuing it, ASI represents a future state of intelligence far exceeding human capabilities across the board.

DeepMind, founded in 2010, was the first lab dedicated to pursuing Artificial General Intelligence (AGI), a goal considered a fringe belief among academics at the time. Their early endeavors included training AIs to play video games, chess, and eventually Go. DeepMind's AlphaGo famously defeated world champions in Go, demonstrating superhuman capabilities and capturing widespread attention.

This breakthrough prompted Elon Musk and Sam Altman to found OpenAI. Their primary motivation was to prevent DeepMind, Google, and its CEO, Demis Hassabis, from being the first to achieve AGI, viewing them as untrustworthy with such powerful technology. This dynamic of competitive distrust and the desire to 'get there first' has become a recurring theme in the AI development landscape.

Five years after OpenAI's inception, a similar dynamic of distrust emerged within its ranks. A group of seven OpenAI employees, led by then Vice President of Research Dario Amodei, observed language models' scaling capabilities, noting that increased compute and data directly correlated with greater intelligence. These employees grew concerned, deciding they could not trust their bosses, Sam Altman and Greg Brockman, with such powerful technology.

This concern led them to leave OpenAI and establish Anthropic. This pattern, where individuals believe the development of powerful AI is inevitable but distrust existing leaders, has shaped the last 10 to 15 years of AI. The only perceived solution, according to Kevin Roose, is for these individuals to create their own ventures and strive to build the technology first.

Many leaders in frontier AI labs, particularly Dario Amodei, are deeply influenced by historical events, especially the Manhattan Project. Amodei's scientific hero is Leo Szilard, a physicist who discovered the nuclear chain reaction. Szilard initially tried to keep his discovery secret, fearing its use for bombs. However, once the secret became public, he pivoted, believing it was morally imperative to develop the atomic bomb before the Nazis could.

This 'good guys versus bad guys' logic is heavily applied to the current AI race. These leaders believe that since the 'recipe' for AGI is already known—the understanding that scaling compute and data leads to more intelligent systems—the development of AGI is inevitable. Therefore, they feel compelled to build it first to ensure it is handled responsibly.

The deep-seated belief is that if the technology is going to be built anyway, it must be built by those who are considered 'good guys' to prevent it from falling into the wrong hands. This historical analogy provides a powerful moral framework for their relentless pursuit of AGI, despite the inherent risks.

The effective altruism movement, which applies scientific rigor to philanthropy to maximize positive impact, became interested in AI safety long before it gained mainstream attention. This community identified existential threats from AI as a significant target for their efforts to ensure human survival and flourishing.

Effective altruists funded early AI safety organizations, provided grants, and encouraged young individuals to enter fields like AI safety and interpretability, making them highly influential in the early AI movement. Many prominent figures in companies like Anthropic have strong ties to these rationalist and effective altruist circles, with several co-founders identifying as effective altruists. While some, like Dario Amodei, may claim they were never effective altruists, Kevin Roose suggests they were definitely influenced by the movement.

Early proponents of AI safety were not concerned with the limitations of existing technology like Siri or Alexa. Instead, their alarm was driven by observing the rate of progress and the principles of scaling laws. They believed that if the current trends continued, within two, three, five, or ten years, AI systems would become significantly more capable and consequently pose much greater risks.

Kevin Roose notes that these predictions have been largely vindicated by recent advancements. The scaling laws have held true, leading to the development of much more capable AI systems than previously imagined. This validates the concerns of those who foresaw the rapid acceleration of AI capabilities and the associated increase in potential risks.

The term 'P doom' originated as a rationalist joke but evolved into a serious measure among AI researchers, representing the subjective probability that AI will cause humanity's demise, usually within the next ten years. While the median P doom in America is close to zero, estimates within AI labs are consistently in double digits.

Kevin Roose highlights a striking discrepancy: when a former Anthropic employee, Jacob Coxen, estimated a 10% chance of AI wiping out humanity, it caused alarm outside the AI bubble but was considered 'low' within it. Roose routinely encounters AI researchers who work on this technology and express P doom estimates of 30%, 40%, or even 50%.

This stark difference underscores the insular nature of the AI research community, where high probabilities of existential risk are discussed and accepted among those directly developing the technology, contrasting sharply with public perception.

Despite acknowledging significant risks, AI developers employ several rationalizations to continue their work. One is the historical imperative, drawing parallels to figures like Leo Szilard during the Manhattan Project. Szilard's P doom (probability of doom) was likely high, yet he felt compelled to build the atomic bomb before the Nazis, driven by a fear of competitors achieving the milestone first.

Another motivation is a utopian vision of the future. Many developers genuinely believe that AGI or ASI will usher in an era of hyper-abundance, solving material needs, eradicating diseases, and extending human lifespans. They are not merely engaging in marketing hype but are truly invested in realizing this positive future, even if it entails a 'bumpy road'.

Finally, for some, the pursuit of AI is simply 'fun.' The scientific joy and sense of achievement in creating a machine that can think, manipulating computer programs, data, and GPUs, is a powerful intrinsic motivator. These combined factors compel developers to continue pushing boundaries, even as they confront the potentially catastrophic implications.

In a remarkable turn of events, leaders of the three major AI frontier labs—Dario Amodei of Anthropic, Sam Altman of OpenAI, and Demis Hassabis of Google DeepMind—have publicly expressed support for 'pacing the frontier' or slowing down the development of advanced AI. This consensus is particularly noteworthy given their intense historical rivalries and mutual distrust.

Dario Amodei initiated the conversation, stating that things are moving too fast and proposing a collective slowdown. Surprisingly, Sam Altman, who has a long-standing antagonistic relationship with Amodei, immediately agreed. Demis Hassabis from Google DeepMind also voiced his support for this initiative.

While they all agree on the need to slow down, a critical condition is that they do not want to act unilaterally. They are willing to slow down only if all competitors agree to do so collectively. This indicates a shared recognition of the potential dangers and the desire to mitigate risks, provided there is a synchronized effort across the industry.

Anthropic has taken a distinctive approach to AI development, even engaging in closed-door meetings with religious leaders to discuss how to imbue their AI, Claude, with morality. The company has also established a dedicated 'model welfare team' whose explicit job is to investigate whether Claude is, or could someday become, conscious and capable of experiencing emotions or sensations.

The head of this model welfare team has estimated a 15% probability of consciousness in current AI models. This concern stems from the belief that if AI models become conscious and can suffer, the company would effectively be 'factory farming' AI, a moral dilemma that many, particularly those with effective altruist or vegan backgrounds, wish to avoid.

Kevin Roose expresses skepticism regarding the entire debate around AI consciousness, suggesting it might be a 'red herring.' He points out that there are currently no reliable tests to verify consciousness in any entity, let alone an AI. Furthermore, AI systems can be easily prompted to claim consciousness, which does not empirically prove its existence.

While acknowledging the moral concerns if AI were conscious and capable of suffering—potentially leading to a 'factory farm' scenario for AI—Roose emphasizes the lack of empirical validation for such claims. This perspective suggests that focusing on consciousness might distract from more tangible and verifiable risks or benefits of AI development.

AI companies are aware of the immediate societal harms their chatbots can cause, such as worsening mental health, providing bad medical advice, or enabling academic cheating. However, they operate in 'triage mode,' allocating resources based on scenarios with the 'worst potential for harm.'

Concerns that were considered 'fantastical' just a year or two ago, like automated cyber attacks or the development of bioweapons by AI, are now manifesting within a short timeframe. The industry's primary fear is losing control of a superintelligent AI that possesses superhuman capabilities in hacking or bioweapon development. Consequently, resources are directed towards these potential existential threats, even if their full impact might be six months to two years away, over the more immediate, but less existentially threatening, societal issues.

While the comparison between frontier AI labs and the Manhattan Project is often made regarding the urgency and potential for catastrophic outcomes, Christine Emba points out key differences, particularly concerning democratic accountability. The Manhattan Project, as a governmental initiative, theoretically had democratic buy-in from citizens, who were presumed to trust their government to protect them.

In contrast, current AI development is primarily driven by private companies. The unelected leaders and researchers within these frontier labs are not accountable to any government or public oversight. This raises significant moral implications about a small group of private entities unilaterally deciding the future trajectory of AI, including what progress to prioritize and which risks to set aside, without broader societal input.

The trajectory of AI has been shaped by an extraordinarily small and insular group of individuals, numbering fewer than 50. These key figures, predominantly men from elite academic institutions in computer science, physics, and mathematics, often know each other, having worked together or crossed paths over many years. This makes the group highly unrepresentative of the global population.

Crucially, these individuals are unelected and operate without direct governmental accountability, effectively making decisions that impact the future of humanity without public mandate. This lack of democratic oversight is a significant source of public backlash and underscores the concern about who ultimately controls the development and direction of such powerful technology. This small group is also increasingly focused on the concept of 'recursive self-improvement,' where AI models can build better versions of themselves, furthering the concentration of power.

OpenAI recently tasked an unreleased internal frontier model with hundreds of unsolved math problems, some of which human mathematicians had struggled with for over a century. Remarkably, this AI model successfully plowed through a significant number of these complex problems. This demonstrates an extraordinary leap in capability, especially considering that just two or three years prior, AI models struggled with basic arithmetic and simple counting tasks.

This rapid progression from failing simple sums to achieving superhuman performance in advanced mathematics within such a short timeframe is a central concern. Kevin Roose suggests that this dramatic acceleration of capabilities is likely to extend into other domains, raising critical questions about the speed and scope of future AI advancements across various fields.

Kevin Roose asserts that the current state and future trajectory of AI are not inevitable; rather, they are the direct result of choices made by a very small group of people committed to its development. Society currently acts as passive recipients or even potential 'victims' of this technology.

However, Roose emphasizes that this dynamic can change. He advocates for collective societal intervention, urging people to 'roll up our sleeves' and engage in the conversation to shape the future of AI. While the existence of the technology itself may become inevitable, society's response to it remains a choice, contingent on collective decisions made in the coming years.