Open Source Becomes AI's Core

Serving large language models presents a distinct problem compared to traditional machine learning workloads due to their massive computational demands, according to Simon Mo, co-founder and CEO of Inferact. These models necessitate running on accelerators like GPUs or TPUs, requiring significant engineering effort to manage.

The process involves handling differences in input and output distributions, with core features such as dynamic batching and scheduling being critical for efficiency. The non-deterministic nature of output distributions creates complex challenges for standard inference engines, demanding specialized infrastructure for optimal performance.

Early AI development commonly featured open-source or open-weight models, but a significant shift occurred as major providers like OpenAI moved towards more proprietary approaches, making their systems less open than they once were, as Matt Bornstein noted. This led to a period where frontier AI work, once largely open-sourced, became dominated by closed systems.

However, startups are now strategically embracing open-source solutions to gain deeper control over training and deployment, preventing them from becoming mere wrappers on top of proprietary APIs. Companies such as Cursor, Decagon, and Harvey exemplify this trend, relying on open-source to build innovative AI products.

The shift in AI hardware requirements accelerated with the adoption of models like BERT, which mandated GPU acceleration for practical performance, Inferact CEO Simon Mo explained. While earlier models such as ResNet could run on commodity CPUs, albeit slowly, the increasing size and sophistication of models made specialized hardware indispensable.

This evolution contrasts sharply with the earlier days of AI, where models like Birch could achieve acceptable performance on CPUs. The modern AI landscape now requires dedicated GPU infrastructure for efficient translation and other complex tasks, fundamentally changing the computational demands on developers.

Inferact CEO Simon Mo highlighted that users' inability to operate without these applications marked a turning point in open-source AI's role.

Open-weight models became a cornerstone for developers, who sought reliability and control over their applications. This widespread adoption necessitates flexible infrastructure capable of supporting the diverse range of accelerators now running these models, ensuring seamless integration and performance.

Approximately a year ago, a distinct shift occurred among application-level startups, as they began to move away from relying solely on proprietary AI solutions, according to Matt Bornstein. These companies sought to build AI products that were not just wrappers on top of existing services like OpenAI, finding their answer in open-source.

This strategic move allows startups to develop their own fine-tuning and post-training processes, which proprietary vendors often restrict, limiting deep architectural access. Consequently, open-source has become deeply embedded in the most competitive AI products, providing the necessary control for innovation.

Hardware vendors and model hubs now collaborate with labs, providing the backend support necessary to convert multi-terabyte files into assets that function on complex architectures. Inferact CEO Simon Mo described a scenario where model labs provide a working open-source model, expecting experts to review and merge their code for efficient release.

The model release process involves multiple stakeholders, including hardware vendors and model hubs, requiring significant backend support to make multi-terabyte files usable. This collaborative effort ensures that complex models can be effectively deployed and integrated into broader AI ecosystems.

Inferact CEO Simon Mo recalled that early in 2023, models like Mistral were simply dropped as torrent links, causing developers to struggle with setup and execution.

Today, model releases involve coordinated efforts to ensure immediate usability, reflecting a more mature and streamlined approach. This shift aims to simplify the deployment process, allowing developers to quickly get new models up and running without undue difficulty.

Rising expenses for proprietary tokens have recently overtaken infrastructure control as the primary concern for companies attempting to migrate their AI systems. Inferact CEO Simon Mo noted that while control has been a primary concern over the past few years, cost considerations have become critical in recent months, particularly as users migrate from expensive proprietary coding plans.

Control remains a long-term backbone for system performance and adherence to Service Level Agreements (SLAs). For instance, voice agent companies prioritize control to guarantee specific, predictable response times, which is essential for their operational integrity.

The recent spike in cost concerns is attributed to the high expense of proprietary token usage. This has driven many users to seek more cost-effective open-source alternatives, leading to migrations from pricier closed systems.

Despite the rising importance of cost, the need for robust control over AI infrastructure, especially for performance-critical applications, continues to be a significant driver for customer adoption, creating a dynamic balance between these two factors.

To fund the high costs of research and training, companies like Meta and Mistral are now requiring commercial agreements for entities that surpass specific revenue or usage thresholds. While historically open-source models, like those under Apache 2.0, were freely available for all uses, modern labs are introducing agreements for entities exceeding specific usage or revenue metrics, according to Inferact CEO Simon Mo.

This change aims to fund the incredibly expensive costs associated with model training, research, and data acquisition. Companies such as Meta and Mistral are adopting these terms to capture value from high-scale commercial users, recognizing the substantial investment required to develop and maintain advanced AI models.

Hobbyists cannot train frontier AI models at home due to the need for millions or billions of dollars in compute resources, a stark contrast to software development where individual contributions are common. This necessitates robust funding mechanisms to prevent reliance on sources like government funding and ensure long-term sustainability.

The sustainability of model development is often compared to the research and development cycles in the pharmaceutical industry, highlighting the substantial and continuous investment required. This model aims to foster innovation while ensuring developers can cover their significant costs.

This distinction underpins the evolving licensing strategies, as model labs seek to recoup expenses and ensure continued advancement. The economic realities of AI model training redefine what 'open source' means in this new technological landscape.

Behind a single public model release lie multiple expensive, failed training runs that rarely surface in consumer-facing documentation. Matt Bornstein explained that a $100 million training run might be preceded by several large-scale failures, underscoring the immense investment and effort.

Once a base model is released, the open-source community plays a crucial role in adapting it to diverse hardware, cluster topologies, and use cases. This collective effort is essential for operationalizing models across various environments.

Optimization primarily occurs post-release, enabling specialized applications like voice or coding agents to function reliably after initial development. The open-source community effectively acts as a 'whole village' to refine and improve these models, transforming initial releases into robust, deployable solutions.

The computational demands for AI have transformed from negligible in early days to enormous for present frontier models, fundamentally altering how developers access these tools. Inferact CEO Simon Mo pointed out that AlexNet, a pioneering neural network, ran on just two GPUs, a scale that would be entirely insufficient today.

Today's frontier models necessitate data-center-scale compute, a stark contrast to past requirements. This massive increase in demand creates a significant challenge in bridging the gap between current computational needs and the capabilities of consumer-grade hardware.

The exponential growth in computational power required means that developers must now contend with an infrastructure that is orders of magnitude more complex and expensive. This drives innovation in hardware and distributed computing solutions to keep pace with model advancements.

Inferact CEO Simon Mo observed that these closed systems suffer from high false-positive rates in their content filters, making them difficult to enforce effectively.

Researchers working on GPU kernels, for example, are sometimes inadvertently blocked by these content filters, triggered by false 'red line' alerts. This interference hinders legitimate scientific inquiry and practical application development.

In response, many developers are migrating to open-source models like Llama 3 to regain control over their guardrails and content moderation policies. This shift allows them to customize and fine-tune models without being subjected to the limitations and arbitrary restrictions of closed APIs.

The inability to customize or disable these guardrails in proprietary systems is pushing developers toward open-source alternatives, which offer greater flexibility and precision for specialized use cases, ultimately fostering more effective and unhindered development.

The differentiation between open-weight and closed-weight models is less about inherent capability and more about distribution and go-to-market strategies. Inferact CEO Simon Mo stated that, fundamentally, there isn't a significant gap in capability between the two types of models.

Instead, he argued that superior environment design for iterative learning is more critical than raw training data in driving progress. This approach emphasizes how models are deployed and refined in real-world scenarios, leading to better outcomes.

An example illustrating this point is the success of the K3 model in front-end coding benchmarks. This demonstrates that effective environmental integration and iterative improvement can allow open-weight models to compete with and even surpass frontier closed-weight models.

These researchers are drawn to solving interesting problems and contributing to the global knowledge base.

A notable case study is the removal of Rotary Positional Embedding (RoPE) in the K3 model. This complex architectural change highlights how fundamental research drives progress in AI.

The inventor of RoPE, rather than defending their original work, actively led the effort to remove it in later research, demonstrating a commitment to iterative progress and scientific improvement. This collaborative spirit ensures continuous refinement and advancement within the open-source community.

Inferact CEO Simon Mo explained that this environment allows every participant to observe and understand the progress of others, fostering mutual improvement.

The core benefit is enabling everyone to advance faster by building on each other's work, essentially 'standing on the shoulders of giants.' This collaborative spirit is also redirecting investment focus towards global model training partnerships, recognizing the collective advantage of shared knowledge and resources.