Large Clusters for Small Models: Challenges and Solutions

These models can typically run on Nvidia hardware that is two to three generations old, allowing them to fit into a single GPU. This accessibility makes them easy and affordable to serve, contrasting with the common misconception that small models inherently lead to performance compromises.

The term "small models" refers to models that can be run efficiently on readily available and affordable hardware, such as Nvidia GPUs from a few generations ago. The entire model fits within one GPU, making deployment simpler. This characteristic stands in contrast to the larger, frontier models that often require more specialized and expensive infrastructure.

There's a noticeable convergence where the diminishing returns seen in larger models allow smaller ones to close the gap. For example, a 27-billion parameter model like Quan 36 is achieving performance comparable to GPT 5.1, making them viable for production if used strategically. It is crucial to use small models for specific tasks, adapting them through evaluations and fine-tuning rather than treating them as generalized, "do-anything" models.

This approach optimizes workloads by assigning specific, highly-trained small models to particular tasks, where they often outperform more generalized large models. Such specialized models, often fine-tuned on extensive datasets, can deliver superior quality for their intended functions.

The open-source ecosystem offers hundreds of thousands of models on platforms like Hugging Face, many with permissive licenses, making them freely available for use. Despite this abundance, the adoption of open-source AI has been slower than anticipated since 2024, with many organizations still relying on proprietary infrastructure like Bedrock. For those venturing into serving small models with open-source infrastructure such as vLLM or SGLang, it’s important to recognize that these tools often require significant customization and research to implement effectively.

Unlike deploying large, single models where the challenges typically involve sharing the model across multiple GPUs and routing requests, small models introduce complexity in balancing local queues due to numerous small requests. This "do-it-yourself" nature, coupled with a lack of pre-tuning for small models, makes achieving high GPU utilization (beyond 20-30%) particularly challenging under constant load.

This inefficiency stems from difficulties in correctly sizing batches due to routing bottlenecks and the diverse nature of requests, such as those involving multiple LoRAs or custom fine-tunes. The process of integrating these adapted models into production often requires extensive back-and-forth between AI engineers and infrastructure teams.

This architecture was the result of extensive iteration and exploration of different cluster designs to address the challenges of model adaptation and diverse environments. The system aims to overcome the limitations of traditional router-based systems by allowing workers to pull tasks from a centralized queue, thereby maximizing saturation and performance.

Superlinked's cluster topology features a gateway that parses incoming requests, attaches metadata, and inserts them into a shared, centralized queue. Unlike traditional routers that pre-decide request destinations, this system allows workers to pull tasks from the queue, enabling better self-saturation. The entire stack, from the control plane down to the GPU runtime, is open source. Additionally, the system employs MessagePack for efficient multimodel data transfer through the API gateway, avoiding the overhead of base64 encoding used in some other APIs.

The API gateway efficiently handles multimodel data by encoding all binary data (such as images or videos) and pushing it through the gateway. This prevents the need for the cluster to access external cloud storage for loading binary data, streamlining the process. The gateway also intelligently separates heavier data components to prevent clogging the internal queue, deferring them to cloud storage "in-flight" while the request is still queued, effectively splitting the request for optimized processing. The entire stack is REST-based, with the gateway and worker communicating via REST, and local runtimes (PyTorch, Kendall, SGLang) attaching over a socket.

A key design principle is to ensure the gateway, as the initial point of contact for requests, performs minimal work to avoid becoming a bottleneck. Instead of fully parsing requests, the gateway quickly examines packet shapes, annotates them, and then directs them to workers. The centralized queuing system allows workers to form their own batches and predict batch costs, improving efficiency. This animation visually illustrates the benefits of centralized queuing, where workers actively pick up tasks rather than relying on a top-down router to fill local queues, which is often inefficient.

Predicting the optimal number of items to pick from a shared queue for an ideal batch size is challenging. The system includes mechanisms to return items to the queue if too much is pulled, despite the network hop this incurs. Superlinked supports three runtimes: PyTorch, Kendall, and SGLang. While PyTorch is robust, Kendall is still a research project, struggling to match PyTorch's performance. SGLang serves as a baseline, and Superlinked has even managed to improve its bare performance by wrapping it with a socket and a RAS sidecar, demonstrating the value of custom optimization.

Kendall offers significantly smaller worker Docker images (about 10% the size of PyTorch's 12GB images), which is advantageous for cold start times and loading on multiple machines. However, achieving comparable performance to PyTorch with Kendall remains a difficult research challenge. Superlinked uses SGLang as a baseline to ensure its systems perform at least as well as established solutions. Wrapping SGLang with a socket and a RAS sidecar has shown improvements over bare SGLang performance, illustrating the benefits of custom optimizations.

The diverse nature of runtimes for small models means it's inadvisable to commit to any single runtime, as custom code for one often won't work with others. Superlinked addresses this by developing a large number of adapters (around 50) parameterized for different models, dealing with the underlying complexity of varied runtimes. This approach allows for maximum performance without compromising latency, a concept referred to as "the knee" in benchmarking—the point of saturation where maximal performance is achieved before latency increases.

Even on relatively small and commodity hardware like the RTX Pro 6000, as well as Nvidia L4, A100s, and H100s, Superlinked’s system can achieve impressive performance. For embedding models, even those with hundreds of millions of parameters, the system can encode hundreds of thousands of tokens per second. These GPUs are readily available on demand in most cloud environments, making high-performance serving accessible.

This approach, considered "low-hanging fruit," offers massive advantages for tasks such as named entity recognition, multi-vector search, and text generation. It allows for quick evaluation of quality with relatively simple infrastructure and a handful of GPUs, providing a powerful alternative for custom AI deployments.

With reasonable infrastructure around the GPUs, linear scaling is achievable with an increasing number of GPUs. A key optimization involves packing multiple small models onto the same GPU, which enhances speed. This strategy combines pinning some models for immediate access with lazy loading and eviction for others, optimizing GPU memory and reducing cold start times. Superlinked also employs auto-research loops to continually add support for new models and improve performance, ensuring ongoing efficiency.

Superlinked's auto-research loops automate the process of adding support for new models and enhancing their performance. These loops include a meta-loop that builds the harness, runs the research, and provides a dashboard for insights. When Superlinked ships support for a model, it comes with all necessary tuning pre-configured, eliminating the need for users to perform parameter sweeps. This approach ensures that small models are not only good, but also cheaper, faster, and smarter, making self-hosting a practical and efficient option.