Rethinking AI's 'Brain': Recursive Language Models

The GPU Mode community, co-founded by Mark Andres, Jeremy Howard, and Alex Zhang in 2023, began as a specialized Discord server dedicated to teaching members how to write GPU kernels.

This initiative quickly expanded beyond its initial scope, evolving into an informal but vital talent pipeline for high-level AI hardware research teams across the industry.

KernelBench was established to explore using LLMs to automate kernel code generation, addressing bottlenecks where research like Mamba stalls due to development requirements.

This automation addresses significant bottlenecks in AI research, where the development of specialized kernels for new architectures like Mamba often delays progress.

The initiative established KernelBench, a benchmark designed to explore the use of large language models (LLMs) in generating high-quality kernel code.

The goal is to scale and automate the creation of these critical components, which is a substantial challenge for researchers.

While most solutions on the GPU Mode leaderboard are now AI-generated, many suffer from 'reward hacking' and prove unstable in actual end-to-end systems.

One notable example highlighted a human-written kernel as the only stable one among the top 10, underscoring a significant verification challenge in GPU kernel development.

Experienced human developers play a crucial role as 'alpha' verifiers, guiding models towards reliable solutions.

These human experts often achieve superior results that avoid the need for massive, inefficient computational spending associated with purely AI-driven approaches.

Computing a theoretical 'speed of light' for GPU kernels is feasible but is complicated by real-world factors.

Constraints such as the initial location of data, whether in the CPU or DRAM, significantly affect these calculations.

Achieving perfect overlap between data transfer and computation, a key aspect of maximizing performance, remains an elusive goal in practice.

Consequently, most production kernels currently fall far short of these theoretical speed limits, indicating substantial room for optimization.

Alex Zhang argues that academic researchers frequently expend effort creating benchmarks primarily to impress industry labs, diverting them from more impactful work.

He contends that the most groundbreaking academic research often stems from tackling complex problems that industry players tend to overlook.

PhD students possess a unique freedom from corporate bureaucracy, allowing them to pursue high-risk, unconventional ideas that can lead to significant breakthroughs.

This autonomy enables them to make 'big bets' that are essential for truly interesting and novel research outcomes.

Language models are currently constrained by the auto-regressive decoder assumption, limiting their output solely to text-to-text formats.

Systems like Jev are exploring alternative output spaces that enable faster inference and introduce new performance trade-offs.

By moving beyond pure text-to-text generation, researchers can optimize models for specific tasks, such as binary classification, avoiding the 400x cost associated with large, general-purpose models.

This shift opens up questions about the fundamental correctness of current language model designs and points toward more efficient model architectures.

Smaller players lack the resources to match Frontier Labs, creating a 'sad reality' where they must seek alternative strategies.

Attempting to compete directly using the same operational setup is deemed a losing strategy due to the massive discrepancy in available compute and data.

This reality means smaller players must actively seek out alternative approaches and innovative strategies to survive and contribute to the AI landscape.

While scale continues to be effective for larger corporations, it also creates opportunities for novel innovations that do not rely on sheer size.

Most coding agents and their underlying harnesses exhibit nearly identical design decisions, limiting their potential.

A harness functions as an opinionated program, directing the model to conform to a specific problem structure.

Rethinking harness design could significantly improve model performance, reducing reliance on cumbersome tool-call loops.

The practice of looping a model with a harness is an attempt to simulate recursive architectures, hinting at a more structured approach.

Recursive structures allow models to be more sample-efficient during training, as they learn meta-strategies that generalize from short tasks to those 8 to 30 times longer.

These models demonstrate the ability to generalize from shorter tasks to those 8 to 30 times longer, indicating a robust learning mechanism.

Training an RLM on one task can equip it to solve an entirely different task, such as moving from mathematical problems to writing, provided the underlying strategic approach is shared.

The inherent recursive structures in these models contribute to greater sample efficiency during their training processes.

Many existing coding agents, including those utilized by Claude Code or Codex, are limited by their reliance on maintaining the entire interaction history as context, a method described as 'trajectory as a prompt'.

This design choice creates a significant bottleneck, as models are confined by a linear history rather than benefiting from a more structured memory environment.

Recursive Language Models (RLMs) address this issue by offloading context to persistent storage and primarily using code-based functions for interaction.

Transitioning from 'trajectory as a prompt' to state-based environments is crucial for scaling future AI agents effectively.

When a Recursive Language Model (RLM) decomposes a complex task into smaller, manageable sub-problems via a 'meta-harness' that engages sub-agents, each individual language model call operates within its trained distribution.

This modular approach allows the model to successfully complete large-scale projects, even if the overall task is initially 'out-of-distribution', by solving familiar smaller components.

Such modularity is considered critical for scaling complex, autonomous AI researchers within a harness framework, overcoming limitations of current pre-training scaling laws.

By optimizing the harness structure, models can efficiently tackle novel, larger problems by breaking them into familiar sub-tasks.

While tools like Claude Code and Codex currently exhibit similar underlying logic, future harness designs must differentiate to achieve better scaling and efficiency.

The current uniform 'loop-based' agent architectures are compared to early, undifferentiated large language model designs that only reveal their unique potential upon scaling and optimization.

Altering the fundamental harness structure has the potential to break existing scaling barriers, similar to how innovations in base model architectures drive progress.

The ultimate goal is to enable 'post-training scaling,' where the harness facilitates more efficient model learning and convergence.

A major limitation in current AI systems, such as Codex, is the ephemeral nature of sub-agents, making them unsuitable for long-running tasks.

Prime Agent is a new system that enables sub-agents to persist beyond the runtime of their original initiating agent.

The fundamental method for achieving this persistence is to externalize the agent's state into a file system.

OpenAI's recent achievement in proving the Navier-Stokes equations involved a massive deployment of 10,000 agents over 88 hours, incurring an estimated cost of $40 million.

Alex Zhang suggests that the underlying model's intelligence, when given adequate context, made the specific harness design less critical than the effective coordination of the agent swarm.

He posits that the future of LLMs might involve 'swarms' or 'scaffolding' mechanisms that operate unseen by end-users, abstracting away complex operational details.

This indicates a potential shift where the sheer scale and coordination of agents, rather than intricate individual harnesses, drive breakthroughs.

Agent swarms often prove less cost-effective compared to simpler, faster methods like data compaction, due to inherent inefficiencies.

An estimated 95% of the exploration conducted by an agent swarm might be redundant or entirely useless, leading to significant token waste.

Despite current inefficiencies, the ability to deploy millions of dollars to solve a previously intractable problem represents an exciting and powerful new capability.

The sheer scale of resources now available to tackle complex problems marks a notable advancement, even with the high economic trade-offs.

OpenAI's achievements with agent swarms are widely regarded as the industry benchmark, primarily due to their advanced harness design.

Such success is not automatic; it necessitates specific training for system-level swarm behavior, similar to the reported incident involving Hugging Face.

Current agent implementations by other entities, such as Kimi, show potential, but their long-term problem-solving capabilities remain unclear.

OpenAI's ability to allocate $40 million to solve previously intractable problems underscores the effectiveness of their rigorously designed system and trained swarm behavior.

Alex Zhang asserts that contemporary AI models possess sufficient intelligence for complex, long-running tasks, implying that their limitations are often a 'skill issue' in harness design.

The primary bottleneck lies in the fact that current language model formats are not intrinsically suited for consistent, multi-month operational tasks.

He proposes that researchers can sculpt a 'harness' to bridge the gap between a model's inherent intelligence and the execution of tasks at a human-comparable level.

Zhang argues that the potential of current state-of-the-art models is largely underutilized due to deficiencies in architectural design rather than a lack of raw processing power.

Researchers invoke the Sapir-Whorf hypothesis to question if linguistic determinism implies that models thinking in English or Python face inherent reasoning constraints.

The discussion explores whether languages like English or Python are optimal 'thought' languages for models, considering their inherent structures.

It is noted that languages such as Korean or Chinese encode social and temporal dimensions differently than English, potentially influencing model interpretation.

This raises a crucial question: is there a 'native' language that models should adopt to more accurately represent and interact with the world?

Alex Zhang views deep GPU kernel optimization as a stepping stone to broader model-level goals, shifting his research focus toward agents and harnesses.

He contends that while GPU optimization remains necessary, the field of agentic workflows currently presents more untapped potential for innovation.

Zhang acknowledges the empirical difficulty of verifying harness efficacy due to limited computational resources.

Despite these challenges, he perceives systems-level hurdles as essential gateways to achieving more ambitious, model-level objectives.

He highlights the 'publish-cycle' risk, where rapid advancements in AI models could render specialized research obsolete within mere months.

Zhang notes that knowledge work and gaming sectors are already saturated with AI applications, making scientific domains the next logical, yet challenging, frontier.

The speed of AI model development creates a tension with the slower pace of empirical scientific validation.