Nebius Optimizes Open LLMs for Production

Nebius operates as a full-stack AI cloud infrastructure provider, managing physical data centers and bare metal capacity to optimize inference performance.

The company's mission focuses on optimizing inference through Token Factory, a strategy supported by its public trading on NASDAQ and a strong partnership with Nvidia to enhance LLM production.

A $2 billion capital injection from Nvidia facilitates the expansion of Nebius' AI infrastructure, with the firm targeting 5 gigawatts of Nvidia systems by 2030.

AI teams often face a dilemma: closed APIs offer ease of use but quickly hit performance ceilings and present black-box operations with linear cost increases.

Conversely, self-hosting provides complete control but demands substantial engineering overhead and extended lead times for deployment.

Token Factory aims to bridge this gap by offering the performance benefits of self-hosting combined with the simplicity and managed aspects of a service, providing a balance between control and ease of use.

Nebius has developed a full-stack AI platform that connects inference, data, and post-training within a single integrated loop.

The platform's inference layer supports over 60 models with dedicated endpoints and tools, the data lab captures production logs for analysis, and the post-training layer enables fine-tuning and optimization based on real-world data.

Nebius supports the use of synthetic datasets for post-training, allowing models to be adapted through methods like LoRA or full fine-tuning to meet specific behavioral expectations.

The platform also offers model distillation, custom speculative decoding, and on-demand quantization and calibration based on user preferences.

By providing a direct flow from inference data collection to production, Nebius eliminates the friction that teams typically face when manually stitching together disparate tools.

Nebius addresses this by enabling a direct and continuous flow from inference data collection to training and back into production, which is crucial for operating a production-grade AI system.

Nebius ensures production-grade deployment by prioritizing engine optimization, vertical hardware control, and enterprise-level compliance.

Optimization involves active tuning of engine-level latency, throughput, and cost for open-source models.

Infrastructure entails vertical hardware stack ownership for superior control, while enterprise readiness addresses security, compliance, and reliability for production deployment.

Teams often encounter difficulties transitioning from initial model deployment to scaling in real-world scenarios.

Nebius emphasizes a full cycle of running, observing, capturing data, and redeploying updates to differentiate teams that build great AI products from those that only ship demos.

Artificial Analysis benchmarks highlight that proprietary and open models now offer equivalent intelligence, reducing the need to constantly chase proprietary options, Sujee Maniyam noted.

Open models, represented in blue on benchmarks, now provide equivalent intelligence to proprietary models, shown in black, often at lower costs, reducing vendor lock-in and tokenomics expenses.

Nebius integrates optimized kernels and NVFP4 support through a collaboration with NVIDIA as part of a multi-layered strategy that extends from hardware-level tuning to automated GPU selection.

At the serving level, internal forks of open-source engines are tailored for specific model performance.

Inference management automates GPU and engine selection, simplifying API usage for customers.

Traditional load balancing is inefficient for LLMs due to varying input/output sizes and fragmented caches, leading to poor inference performance in large-scale GPU clusters.

Nebius employs cache-aware routers that track stored cache locations, improving coherence and directing requests to GPUs with relevant cached data.

This approach maximizes inference speed by leveraging existing cached data rather than generating tokens from scratch.

Junior-senior architectural patterns resolve the slow, sequential token generation of large language models by having a smaller model draft content for a larger model to verify.

This technique uses a smaller, faster "junior" model to generate the bulk of tokens, which a larger "senior" model then verifies, only regenerating tokens if the draft is incorrect.

Mixing small and large models through speculative decoding offers significantly better performance than relying solely on large models.

The Nebius Token Factory platform automates draft model training, and training these models on custom production data can yield performance improvements beyond the 30% seen with generic data.

Caching identical tokens to prevent regeneration enables speed improvements of 5x to 10x, with system-level memory offloading managing the associated increase in consumption.

To manage high memory consumption, the system automatically offloads the cache from GPU memory to regular memory.

This process occurs without any manual user intervention, ensuring efficiency and ease of use.

Decoupling the compute-intensive prefill phase from the memory-intensive decoding phase allows Token Factory to distribute workloads across distinct GPU sets while sharing the KV cache to mitigate hardware resource contention.

Token Factory separates these phases onto different GPU sets, allowing one set to handle prefill and another to handle decoding, sharing the KV cache between them for increased efficiency.

Quantization reduces numerical precision to enhance model efficiency, but excessive quantization can degrade model quality.

The platform conducts extensive experiments to determine the optimal 'sweet spot' for quantization, ensuring efficiency is maximized without sacrificing performance.

Quantization reduces precision to improve efficiency, but over-quantizing can lead to model quality degradation.

The platform performs extensive experiments to find the optimal quantization level, allowing models to run efficiently without significant performance loss.

Cache-aware routing is crucial for maximizing performance, especially when managing cached data across GPU clusters.

Separating prefill and decode phases is also essential for large models, such as those with 1 trillion parameters, with most of these optimizations operating seamlessly behind API-based services.