CoreWeave Details Vertical Mobility for AI Inference

Sitanshu Gupta, who leads inference at CoreWeave, introduced the company's strategy for vertical mobility, emphasizing the development of an inference platform capable of serving models from small to trillion-parameter scales. He highlighted his background from AWS Annapurna Labs and Habana Labs as foundational to this effort. The core philosophy is to minimize platform reconfigurations while continuously enhancing existing infrastructure to meet evolving demands.

The serverless model allows customers to pay per token without needing to manage hardware, clusters, or orchestration, with CoreWeave handling all infrastructure. In contrast, the dedicated model provides users with greater control over hardware selection and model deployment, offering specific configurations to optimize performance.

To mitigate performance degradation often seen in standard serverless environments due to shared capacity, CoreWeave introduced a 'provisioned throughput' feature. This allows customers to inform CoreWeave of their traffic profiles, enabling the platform to allocate dedicated resources for them. This guarantees Service Level Agreements (SLAs) without customers needing to manage the underlying infrastructure.

CoreWeave distinguishes between several workload types—agentic, chat, voice/video, and batch—each with unique latency and throughput requirements. Agentic and chat workloads typically feature high input and low output sequence lengths, but agentic interactions demand significantly lower latency for multi-turn conversations than chat. Real-time voice and video applications are strictly latency-sensitive, requiring immediate processing. Conversely, batch workloads have high tolerance for latency, often taking minutes or hours to complete, which allows for more flexible scheduling and resource allocation.

CoreWeave allows customers to optimize capacity by repurposing dedicated inference resources for batch jobs during off-peak hours. For example, a customer using capacity for real-time workloads during the day can schedule batch workloads on the same capacity from evening to night. This API-driven scheduling enables automated scaling up and down based on time-of-day demands, maximizing resource utilization during overnight windows.

Optimizing Key-Value (KV) cache is crucial for agentic workflows, as prefill is identified as the most expensive phase in the inference process. Reusing the KV cache across multiple turns in agentic interactions significantly reduces compute time. CoreWeave addresses this by offloading the KV cache to high-bandwidth storage, preventing unnecessary re-computation and allowing rapid reloading into High Bandwidth Memory (HBM) when accompanying requests return, using techniques like LM cache and Mooncake.

CoreWeave leverages quantization to NVFP4 and speculative decoding as primary methods to enhance throughput and reduce latency. The company offers custom speculative training as an asynchronous service, allowing customers to train speculators with their specific datasets for better acceptance rates. This approach aims to significantly increase output throughput without impacting live inference performance, ultimately providing superior price-performance benefits.

The company quickly rose on the Gemini 2.6 and 2.7 leaderboards, according to Artificial Analysis. Furthermore, OpenRouter data, reflecting actual user traffic, demonstrates that CoreWeave's deployment speed is comparable to leading providers such as Fireworks, emphasizing the use of real-world benchmarks alongside specific workload testing.