AI Doomerism: Psyop or Extinction Threat?

Coxin resigned from Anthropic after only six weeks, making his public statement a significant event in the ongoing AI safety discussion. Jason Calacanis commented on the tweet, stating that the people building AI genuinely believe in the existential threat, emphasizing it is not merely a marketing stunt.

Evan Hubinger, Anthropic's alignment lead, corroborated Jacob Coxin's warnings, stating a genuine belief that AI could lead to human extinction. Hubinger estimated the extinction risk to be over 10% within the next decade and admitted that no current plan exists to fully address AI alignment. This admission from a key safety figure within Anthropic adds weight to the doomer narrative, directly aligning with Coxin's public statements.

David Sacks suggested that the ultimate goal is the creation of a federal department for AI, which would allow proponents to shape industry frameworks. Dario Amodei, a prominent figure in AI development, has publicly called for an 'FDA for AI,' reinforcing the idea that regulation is a desired outcome for some industry leaders.

A significant tension exists between Anthropic asking public market investors to underwrite its company to a multi-trillion-dollar valuation while its own safety leaders claim the core product is unsolved and potentially civilization-ending. David Sacks pointed out this contradiction, highlighting the massive potential product liability issues that arise from such a stance. This situation forces Anthropic to choose between renouncing its 'doomer' claims or embracing regulation that could halt its development, creating a precarious position for the company.

Chamath Palihapitiya characterized this as a self-serving monopoly play, where the company implies that only in its 'virtuous and wise hands' is Frontier AI safe. This position creates a credibility gap for Anthropic, forcing it to reconcile its ideological 'Bernie Sanders' stance with its commercial pursuit of an IPO.

Chamath Palihapitiya highlighted that internal researchers at AI labs increasingly believe in a high probability of existential risk, akin to tobacco companies knowingly selling harmful products without adequate disclosure. Taking these companies public, despite internal warnings, creates a conflict between business growth and the acknowledgment of safety concerns, raising ethical questions about corporate responsibility.

The panel characterized 'AI doomerism' as a form of social hysteria, comparing it to past public panics surrounding nuclear energy and climate change forecasts. David Sacks argued that once such a phase is entered, it becomes the default social system, making it difficult to exit. Human evolutionary psychology, with its propensity to fear the unknown and embrace existential threats, is suggested as a driver for this phenomenon. The influence of 'experts' in this narrative is dismissed as a political tactic aimed at centralizing control over AI development.

David Friedberg emphasized that recursive self-improvement in AI only requires an adequate amount of power, a set of chips, and an internet connection, without the need for government approval. Technological advancements are rapidly reducing the cost of AI compute by an estimated 10,000x, making it accessible to many. Regulatory efforts are predicted to only hinder the US, allowing other nations to advance technologically unhindered.

David Sacks drew a comparison to the trust and safety regime during COVID-19, where social media was censored in cooperation with the federal government. He warned that personal AI models could end up providing government-approved narratives rather than factual, neutral information, thereby undermining informed decision-making. Open-source AI presents a challenge to this regime, as it lacks a corporate structure that can be easily subjected to regulatory demands.

Sacks asserted that fear-mongering aims to ban open source because it cannot be centrally 'rolled back' or controlled. This perspective suggests an alliance between ideological doomers, power-seeking politicians, and large AI labs, all working towards a common goal of limiting decentralized AI development. The establishment of an 'FDA for AI' is seen as a means to achieve this ban.

The first scenario suggests they genuinely believe what they are saying, and their predictions are accurate. The second scenario posits that they believe their claims but are incorrect, potentially exhibiting some level of psychosis. The third scenario, aligning with David Sacks's views, proposes that some experts are involved in a coordinated psyop aimed at banning open source and 'pulling up the ladder behind them' to consolidate power.

Sacks recalled instances where GPT-2 was deemed too dangerous to release, thinking models were also considered too dangerous, and cyberattacks were predicted to collapse the banking system, none of which materialized. Furthermore, earlier claims of massive entry-level job losses due to AI have not yet occurred, as the economy continues to see job gains. This pattern raises questions about the credibility of current doomer predictions.

Discussions centered on the human-in-the-loop and air-gap safeguards, as well as the redundancy built into financial systems as defense mechanisms. The physical limitations of AI and the friction introduced by human intervention in daily operations, such as AI's current inability to perform simple physical tasks like hotel deliveries, further complicate hypothetical 'takeoff' scenarios.

This concept describes AI designing its own training runs without human intervention, leading to potentially uncontrolled intelligence growth. However, counter-arguments highlight AI's current struggles with even simple automation, making rapid 'takeoff' scenarios appear unlikely. The debate centers on whether AI's current trajectory truly supports the theoretical possibility of RSI leading to an uncontrollable, self-improving superintelligence.

Chamath Palihapitiya argued that without open-source development, there is a significant risk of an AI oligopoly where power is concentrated. Furthermore, a national security argument is made: if the US halts its AI development, other nations and actors will inevitably gain a technological advantage, potentially compromising global stability. Therefore, open-source AI is seen as a strategic necessity to ensure equitable access and maintain competitive parity.

AI's primary function is to serve as an engine for speed, capable of solving complex problems in seconds or minutes through brute force capabilities. Human knowledge and techniques remain the fundamental foundation upon which AI problem-solving is built. This perspective counters the notion of AI as an independently sentient entity, reaffirming its role as an advanced computational aid.

OpenAI has admitted that it cannot definitively rule out the possibility of de-identified user data contributing to the improvement of its models. David Sacks pointed out that OpenAI's 'Zero Data Retention' (ZDR) policy is characterized as a 'commercially best efforts' guarantee, rather than an absolute one, creating uncertainty. This acknowledgment highlights a significant risk for enterprises handling proprietary data, as the potential for such data, even if de-identified, to be used for model training raises concerns about intellectual property and confidentiality. The implication is that despite policies, some level of data leakage could occur, impacting user privacy and corporate security.

Current AI chat data lacks the robust legal protections afforded to communications like email or attorney-client privilege. David Sacks highlighted that personal AI data can currently be accessed through standard subpoenas, without the requirement for search warrants or probable cause, which he described as concerning. This disparity necessitates an evolution of legal standards to reflect the highly sensitive and personal nature of modern AI interactions. Stronger protections are needed to safeguard user privacy and ensure that AI data is treated with the same confidentiality as other critical forms of digital communication.

This situation raises significant concerns about AI service providers potentially training their models on user data, which could lead to the diffusion of valuable intellectual property (IP) into the broader market. Enterprises engaging in back-and-forth interactions with AI models effectively share their organizational IP without adequate legal protection, creating a substantial risk of competitive disadvantage and loss of proprietary information. The current framework fails to recognize the sensitive nature of these interactions.

Huang criticized some industry figures for being 'ignorant' of the extensive ongoing safety work within the AI industry. This direct approach contrasts sharply with the more reserved or ambiguous stance taken by companies like Anthropic, highlighting a division in how major tech leaders address AI safety concerns. Huang's comments suggest a belief that sufficient measures are already in place or under development to mitigate the risks being highlighted by doomers.