KV Cache-Aware Routing and P/D Disaggregation on Kubernetes

This shift from considering latency as a secondary or afterthought metric emphasizes the necessity of prefix routing coordination. Such coordination is essential for efficient scheduling and ensuring that large language model inference operates with minimal delay.

It is essential to measure KV cache throughput separately due to the significant economic stakes involved. Anthropic API pricing, for example, clearly shows a tenfold cost difference between cached and non-cached tokens. This substantial disparity means that efficient management and tracking of the KV cache directly impact the financial viability of business models relying on large language model services.

These EP plugins actively probe pods to assess their current status, including running or waiting requests and the availability of their KV cache. The routing logic then prioritizes pods with the lowest load and the highest probability of a cache hit, ensuring efficient resource utilization and reduced latency.

Current efforts focus on implementing more sophisticated KV cache management, including offloading tiers using NVMe SSDs and file system XF. This also involves integrating KV-centric stores like Mooncake to enhance storage efficiency. Furthermore, smarter session-aware eviction policies, such as priority-based eviction and session pinning, are being implemented to optimize cache utilization and performance. These techniques aim to manage hot, warm, and cold cache tiers effectively.

When the very first request is sent, it populates the KV cache and takes approximately three seconds to process. However, if a subsequent request reuses the same system prompt, thereby hitting the KV cache, the processing time significantly reduces to about one second. Conversely, changing the system prompt results in a three-second delay, indicating that a cache hit did not occur at a different pod address, underscoring the efficiency gains from cache reuse.

While prefix routing and KV cache routing effectively address Time To First Token (TTFT) issues and improve overall throughput, agentic workloads demand more. For these complex tasks, the primary concern shifts beyond TTFT to optimizing inter-token latency. This requires specialized solutions, specifically prefill/decode disaggregation, to manage the intricate timing requirements between token generations.

It offers a unified control plane specifically designed for the agentic era, facilitating complex inference workloads. The architecture utilizes workload APIs, such as leader-worker sets, to orchestrate multi-node model execution. Additionally, intelligent autoscalers continuously monitor real-time traffic, enabling independent scaling of pods to adapt to varying demands.

In traditional aggregated serving, a single pod is responsible for optimizing both Time To First Token (TTFT) and inter-token latencies. However, Prefill/Decode (P/D) disaggregation fundamentally changes this by allowing the prefill and decode phases to be independently scalable. This decoupling addresses performance bottlenecks by assigning specialized resources to each phase, leading to more efficient and responsive LLM inference.

Collocating both prefill and decode tasks on the same GPU creates an issue known as phase interference, which significantly hinders performance. Prefill tasks are computationally intensive and benefit from large batch parallelism, whereas decode tasks are memory-bandwidth hungry and highly sensitive to latency. When these distinct tasks share the same GPU, they interfere with each other, leading to performance stalls and unpredictable jitter in execution. This conflict underscores the need for specialized handling to optimize each phase independently.

The prefill worker is responsible for constructing the initial KV cache, while the decode worker subsequently pulls the necessary KV cache data via the network fabric. This coordinated process ensures efficient task distribution and specialized handling for each phase.

Experimental results demonstrate a substantial performance improvement with P/D disaggregation compared to standard aggregated deployment. In the aggregated deployment, the P99 Inter-Token Latency (ITL) fluctuates around 900 milliseconds, with noticeable variations. Conversely, the P/D deployment achieves a P99 ITL of approximately 100 milliseconds, representing a nearly nine-fold improvement. Furthermore, the P/D results exhibit significantly smoother and more stable performance, indicating greater consistency and reliability.

Prefill/Decode (P/D) disaggregation particularly shines in the middle of the concurrency regime. While standard Kubernetes scheduling serves as a baseline, KV cache-aware routing alone offers performance improvements over standard aggregation. However, it is specifically within this mid-range of concurrency that P/D disaggregation demonstrates its most significant advantages, providing superior efficiency and latency reductions compared to other methods.

It is best suited for managing long contexts, especially with high input-to-output ratios, where the distinct phases benefit from separate optimization. The approach requires large models capable of complex model parallelism to fully leverage its advantages. Furthermore, P/D disaggregation is ideal for use cases with strict inter-token latency (ITL) requirements, where minimizing the delay between token generations is critical.

Without such robust network infrastructure, the benefits of disaggregation cannot be fully realized, and aggregated serving might be a more suitable option. The decision to adopt disaggregated serving is also influenced by factors such as context length, model size, and concurrency, which collectively determine the feasibility and effectiveness of implementing specialized network solutions.

The scheduler must continuously evaluate Service Level Objective (SLO) targets, Quality of Experience Parameters (QEPs), KV cache locality metrics, Prefill/Decode (P/D) ratios, and network topologies to route requests optimally. Dynamic P/D rate matching is crucial as traffic patterns evolve, requiring constant adjustment to maintain efficiency. Independent scaling of P/D pools is also vital for resource optimization, ensuring that prefill and decode stages can be provisioned according to their specific demands. Furthermore, platform architects must constantly adjust model parallelism strategies to adapt to varying workloads and model complexities, encompassing techniques such as pipeline, tensor, and data parallelism. This intricate interplay of factors necessitates a flexible and adaptive architectural design to achieve high performance and cost-effectiveness in LLM serving.

These workloads are characterized by heavy prefill stages, often referred to as a "wild path" in LLMD. The design incorporated different parallelism strategies, allowing for independent scaling of prefill paths. Specifically, the prefill pool was configured with up to three workers, optimized for high throughput to handle the intensive prefill requirements. For the decode stage, one dedicated worker was utilized, specifically optimized for low latency. This modular architecture enables throughput scaling without needing to reconfigure the decode workers, providing flexibility and efficiency. Additionally, experiments revealed that BF16 KV cache performed faster than FP8 for longer prefill operations, contributing to overall performance gains in these complex agentic scenarios.

For agentic workload datasets, which exhibit a high prefill-to-output ratio of 45 to 1, prefill often serves as the primary constraint. By employing a configuration with two prefill workers and one decode worker (2P, 1D), significant performance improvements were observed, including a fourfold acceleration in Time To First Token (TTFT) and a 60% increase in processed requests. Looking ahead, the focus is shifting towards enhancing session graph orchestration and improving state reuse within the LLMD framework. The ongoing open-source development of LLMD involves key ecosystem partners such as Google, IBM, and NVIDIA, fostering collaborative innovation in this rapidly evolving field.