Fractile Bets on High-Speed AI Inference Chips

Founded in mid-2022, Fractile focuses on high-speed inference to enable models to run faster while supporting long-context processing through 'test-time compute'.

The company's core objective is to allow models to operate at significantly higher speeds than existing chips, all while fully supporting the demands of long-context processing.

This architectural strategy prioritizes 'test-time compute,' a method designed to scale model performance efficiently, drawing parallels to the operational scaling observed in systems like AlphaGo.

Many contemporary AI chips, including those from hyperscalers like Google's TPU, Meta's MTIA, and Microsoft's Maya, alongside offerings from various companies, predominantly utilize standard industry components.

These chips frequently share common elements such as High-Bandwidth Memory (HBM), tensor cores optimized for matrix multiplication, and advanced packaging techniques from TSMC.

A considerable number of proprietary chip initiatives are executed through partnerships with ASIC houses like Broadcom, resulting in a diverse yet fundamentally similar array of hardware designs across the industry.

Few teams currently possess the comprehensive expertise required to navigate the entire silicon stack, encompassing everything from initial architectural design to physical implementation and subsequent interactions with foundries.

Most companies opt to delegate the complex backend physical design process to specialized ASIC houses, limiting their direct control over the full development lifecycle.

Fractile distinguishes itself by maintaining internal control over both the architectural and physical design layers, enabling a more integrated and customized approach to chip development.

By aligning hardware with the massive volume of matrix multiplications found in AI models, such as the development of the tensor core, architects translate abstract model behavior into optimized circuits.

The creation of the 'tensor core' exemplifies how architects optimize hardware, in this case for the massive volume of matrix multiplications inherent in AI models, demonstrating the need for deep hardware understanding combined with insights into AI model behavior.

Fractile operates with a team of approximately 150 employees, encompassing front-end designers, physical design specialists, and a dedicated backend implementation team.

This in-house control over advanced packaging and design processes fosters a more agile and closed-loop development cycle, reducing reliance on external partners.

The integrated structure mitigates the risks typically associated with outsourcing complex design stages, ensuring greater consistency and efficiency from concept to deployment.

The company identified that the marginal costs associated with model deployment represent an ongoing expense incurred with every single interaction or query.

This early strategic decision led to prioritizing speed as a fundamental architectural driver in their chip designs.

Fractile initially utilized SRAM-based chips to exploit the high bandwidth achievable between compute units and memory.

SRAM technology facilitated processing thousands of tokens per second by maintaining weights and KV caches in close proximity to the compute logic, enhancing operational efficiency.

However, concerns regarding the scalability of this approach emerged towards the end of 2023 and into 2024, driven by the rapid growth in model parameters and context lengths, prompting a re-evaluation of their memory strategy.

Fractile is actively collaborating with memory vendors and logic foundry partners to secure extremely high-bandwidth access to higher-capacity DRAM.

The team is strategically transitioning away from SRAM to DRAM to overcome capacity limitations while striving to maintain the necessary operational speed, addressing prior scalability concerns.

This integrated approach is projected to yield a new platform that will begin ramping up production and deployment in the second half of the coming year.

Scaling inference at a data center level requires minimizing the cost per gigabyte, leading Fractile to prioritize high bandwidth from low-cost DRAM as a foundational building block.

To scale inference effectively at the data center level, it is crucial to minimize the cost per gigabyte of memory, as this directly impacts the economic viability of large-scale AI deployments.

Fractile views the development of a pathway to achieve aggressive bandwidth from low-cost DRAM as a fundamental building block essential for the future of AI infrastructure.

AI models undergo significant evolution every few weeks, creating a challenge for traditional chip delivery cycles to keep pace.

Chip design is guided by recurring demands across models, such as the need for increased memory bandwidth and efficient low-batch autoregressive generation.

Fractile is developing AI-forward design processes to manage the entire end-to-end stack, enabling quicker responses to advancements in attention mechanisms and sparsity.

Amdahl's law provides a relevant analogy for organizational bottlenecks that persist within the chip design process.

Traditional front-end chip design can typically span approximately 12 months, contributing to lengthy development cycles.

Despite potential design accelerations, industry-wide bottlenecks, such as fab cycle times, continue to present a constraint, with production still requiring 3 to 5 months.

NVIDIA's current systems feature complex ecosystems, often comprising 6 to 9 custom chips, all developed internally.

Increasing productivity in chip design allows companies to take more 'shots on goal,' enabling a continuous stream of innovative bets.

Achieving a structural advantage of 3 to 6 months in the chip development cycle can serve as a massive competitive differentiator in the current market, ensuring dominance in deployments.

A prominent semiconductor CEO projected a 10-year timeline for achieving full intent-to-GDS2 file automation in chip design.

Goodwin proposes a heuristic of dividing conventional timelines by four, suggesting that end-to-end AI-driven prototyping could emerge within the next few years, significantly compressing development schedules.

Representing an intrinsic latency in the workflow, conventional place and route tools often require days to generate GDS2 files from initial architect designs.

These traditional algorithmic approaches introduce inherent latency into the workflow, making the transition from architectural intent to final GDS2 files a computationally intensive and time-consuming process.

As AI intelligence advances, the experiment itself frequently becomes the primary bottleneck in the development process.

This necessitates a commitment to extensive, human-equivalent reasoning and deliberation before initiating any potentially costly experiment.

Fractile aims to accelerate the resolution of complex problems by facilitating deeper reasoning tokens prior to the physical placement of circuits, effectively optimizing the design workflow.

The architectural design supports current auto-regressive transformers while actively exploring new paradigms for model design, including novel scaling laws that extend beyond traditional 'flop scaling' to incorporate 'bandwidth scaling.'

Fractile's chips are engineered to deliver 25 times more bandwidth per chip compared to existing HBM-based chips, aiming to enable a new landscape of model architecture.

Mixture of Experts (MoE) models achieve superior performance when their sparsity is increased, for instance, from 1 in 6 to 1 in 128 or 1 in 256.

However, current HBM-based GPUs and XPUs often encounter significant bandwidth bottlenecks when attempting to efficiently serve these sparser models.

Inference on such models typically results in very low Memory Flops Utilization (MFU), indicating underutilization of computational resources due to memory constraints.

Over the past two decades, memory bandwidth has scaled at a significantly slower rate (40x) compared to raw compute/flops (1,000,000x), making improvements in memory bandwidth crucial for reducing the computational power needed to achieve a given level of model intelligence.

Relying on common platforms mitigates the risk of being locked into a single, unproven hardware ecosystem, which frontier AI labs currently view as an existential threat.

If a competitor develops a computational breakthrough tied to a different hardware architecture, a lab could face a substantial 9-month lag in deploying new models, posing a significant competitive disadvantage.

Standardizing on common platforms helps mitigate the risk of becoming trapped within a sub-optimal hardware ecosystem, allowing labs to focus their competitive efforts on middleware and model weights.

Third-party chip providers, like Fractile, aim to align with the specific needs of frontier workloads, offering specialized hardware solutions without forcing labs into risky, proprietary hardware silos.