Sail Research Aims to Cut AI Token Costs by 1,000x

The market's focus has decisively shifted from low-latency, real-time AI interactions to long-horizon, persistent tasks. Historically, AI development was geared towards immediate, cursor-driven responses. However, as of six months ago, there has been a clear realization that agents are more effective when performing longer, proactive tasks in the background. This shift means that rapid token output is no longer the primary requirement, opening doors for extreme efficiency optimizations.

Latency becomes a bottleneck primarily when a human is actively waiting for an AI agent's response, or 'in the loop.' The goal for Sail Research is to move agents to a 'human time scale,' where tasks are completed proactively and in the background, eliminating the need for immediate human interaction. Neil Movva argues that the best latency is effectively zero latency because the task is finished before the human even considers asking for it, allowing seamless integration into workflows without interruption.

The theory that giving an agent more time leads to better answers, known as test-time compute scaling, has been proven viable. This concept, theorized about two years ago, became practical with the advent of Opus 4 late last year, enabling agents to run effectively for an hour or more. Evidence now strongly suggests an exponential growth in the capabilities of AI models to handle increasingly longer tasks, validating the long-horizon strategy.

Neil Movva predicts a significant shift in AI workload distribution, moving from the current 50/50 split between background and real-time tasks to a 90/10 split favoring background agents. This transition is driven by the understanding that background agents can bypass the limitations of human attention spans, allowing for more persistent and proactive work. The long-term vision positions background agents as the dominant force in AI utility, transforming how AI is integrated into daily operations.

During the early development of tensor cores at Nvidia, AI researchers faced intense internal competition for silicon die area with the established graphics teams. This allocation was initially limited, with AI features receiving only about 5-10% of the total die area. Every design choice involved significant opportunity costs, leading chip designers to carefully guard their allotted space due to its inherent value and scarcity, highlighting the early struggles to integrate AI-specific hardware.

Sharding tasks across multiple GPUs introduces communication overhead, preventing linear scaling of speed with increased hardware. For example, using eight times more hardware typically yields only about four to five times the speed, not eight times. This sublinear scaling means that as work is partitioned into smaller tiles, the efficiency decreases, underscoring the challenges of achieving perfect parallel processing. Engineers must find ways to minimize this overhead to maximize hardware utility.

A hybrid approach is emerging as essential for optimizing language models, pairing high-speed memory chips, like those from Cerebras, with high-capacity chips such as GPUs. This strategy addresses the dynamic growth of the Key-Value (KV) cache, which expands based on user interaction and poses a significant bottleneck. Unlike model weights, which are static, the KV cache's fluctuating demands necessitate a flexible memory solution that combines both rapid access and substantial storage capacity to maintain performance.

The current transformer architecture has a fundamental inefficiency, which Neil Movva calls "the original sin of transformers": juxtaposing memory-bound attention layers next to compute-bound Multi-Layer Perceptron (MLP) layers. Modern chips struggle to be simultaneously efficient at both memory operations and complex computations. Moving forward, both Nvidia and Grock are likely to adopt a division of labor, optimizing specialized hardware for each type of layer, which could significantly boost performance and efficiency by addressing this core architectural challenge.

The next phase of AI data development will move beyond internet data, which is largely exhausted for model training, towards model self-improvement through Reinforcement Learning (RL) environment gyms. Expert-level feedback and verifiable environments (such as coding and mathematics) are now more valuable than random user interactions. This approach allows models to refine their capabilities in structured settings, leading to more robust and accurate AI, marking a significant evolution in data acquisition and utilization.

Specializing exclusively in kernel engineering is not considered the most meaningful long-term competitive advantage. Instead, Neil Movva argues that advancements in kernel optimization, like those seen with Mythos or GB 5.6, are becoming standard capabilities that benefit everyone. Frontier model capabilities, particularly in code generation, are evolving into a shared baseline across the industry. Therefore, while crucial for performance, kernel engineering improvements lift all boats rather than offering an exclusive edge.

Unlike historical hardware booms, such as the dot-com era, which were driven by speculative infrastructure investments, AI inference consumption is immediately valuable and non-speculative. Tokens are purchased for immediate use, not hoarded, indicating real-world demand. This immediate utility has led to inference spending hitting enterprise cloud caps, demonstrating that the market is driven by actual consumption rather than anticipated future needs, ensuring sustained growth in demand for AI resources.

Inference workloads are increasingly suited for distributed 1-megawatt data centers, offering a viable alternative to the massive, centralized power requirements of large training clusters. Building these smaller, distributed facilities is more feasible given the growing challenges of securing concentrated power for traditional data centers. Liquid cooling technology plays a crucial role, enabling high power density by packing a megawatt into roughly eight racks, making efficient use of space and resources.

Sail Research is adopting a radical approach to achieve unbeatable economics by deliberately sacrificing traditional data center redundancies. This includes eliminating backup diesel generators, removing redundant networking, and operating with single-source power and fiber lines. The strategy involves accepting a 95% uptime, a trade-off considered acceptable for long-horizon background AI agents that do not require continuous, instantaneous availability. This aggressive cost-cutting measures are central to their mission of drastically reducing AI token costs.

Neil Movva emphasizes that curiosity and a 'love for performance' are the most crucial traits sought in performance engineers, rather than specific AI experience. He notes that CUDA or GPU concepts evolve so rapidly that long-term specialized experience is less relevant than fundamental curiosity and the drive to deeply understand how machines operate at a microsecond level. This innate passion for optimizing performance cannot be taught and is critical for pushing the boundaries of efficiency in AI infrastructure.

Sail Research's ultimate mission is to reduce AI token costs by three to six orders of magnitude, making them as cheap as humanly possible. This ambitious goal aims to transform AI from an expensive consultant model into a ubiquitous, customized agent system accessible to everyone. The strategy involves leveraging every layer of the AI stack, including unconventional power sources, diverse chip architectures, and available land across the United States to maximize supply-side levers and achieve unprecedented cost efficiency.

New hardware entrepreneurs must identify and address the critical bottlenecks in the modern chip supply chain to succeed. These include TSMC wafer capacity, HBM capacity, advanced packaging, and power supply. Success hinges on understanding where Nvidia, the current market leader, might be 'spiky' or under-optimized, and then strategically pushing in those specific, underpriced directions. This focused approach on overcoming supply chain limitations is essential for new entrants to carve out a competitive niche.