OpenAI Shifts LLM Routing to Explicit Policies

The talk outlined the system architecture, specifically the control and data planes, while emphasizing the move toward a policy driven by engine-reported signals.

Its primary responsibilities involve selecting the appropriate engine for a request and then proxying that request, with a particular focus on distributing requests across regions or continents to enhance resilience against localized failures.

TTFT and token throughput metrics are combined with KV cache awareness to ensure efficient routing and avoid costly recomputations.

Crucially, the system needs to be aware of the KV cache to prevent recomputation and improve latency, often by maintaining state for follow-up requests and routing them to the same engine to ensure efficiency in a system balancing performance, reliability, and locality constraints.

OpenAI's initial routing system employed weighted consistent hashing, taking into account engine capabilities and compute constraints.

Engine weights were determined through a periodic feedback loop that assessed performance scores against fleet averages, with a controller adjusting weights upward for high-performing engines and downward for underperforming ones.

This mechanism functioned similarly to a proportional (P) controller in control theory, continuously guiding the system toward a desired operational state.

Automated load balancing was achieved by feeding engine signals back into the system, which decreased the reliance on manual intervention.

Complexity arose from these integrated signals, making individual routing decisions difficult to interpret and creating side effects during parameter fine-tuning.

Inconsistent characteristics among different GPUs across various engines resulted in uneven load distribution within the system.

The feedback loop exacerbated this issue by creating oscillations, where traffic would repeatedly shift between a few engines, disrupting KV cache utilization and ultimately degrading overall performance.

Simple round robin load balancing proved ineffective because inference engines are not homogeneous in capacity, health, or geographic distance, and such an approach also disrupts cache locality for related requests.

Independent routing by multiple local CPU clusters could lead to an engine being overloaded while others remained underutilized across the global system, illustrating the limitations of naive distribution strategies.

OpenAI proposed an architecture featuring a global control plane to compute globally optimized routing weights, providing a comprehensive view of all CPU clusters and GPU engines.

Operating via asynchronously refreshed weights, the local data plane incorporates TTFT and TBOT real-time signals and offline capacity regressions to maintain routing efficiency.

Three distinct paths define this architecture: the flow of inference requests, real-time engine signal reporting, and the asynchronous updates of routing weights.

The data plane manages the first pass of routing synchronously and locally within the CPU cluster, ensuring speed.

Two additional asynchronous loops are then used to compute routing decisions, allowing for long-term optimization without impeding the immediate request flow.

Traffic demand and GPU capacity are not uniformly distributed geographically, rendering nearest-engine routing an inefficient strategy.

For example, if CPU cluster B sends 120 requests per second (RPS) to an engine B with only 100 RPS capacity, offloading 20 RPS to an underutilized engine C, even if geographically more distant but using only 40 RPS of its 80 RPS capacity, would improve total end-to-end latency.

The objective function combines network transit and engine processing times to achieve this goal.

The system must route all traffic demand, respect the capacity limits of each engine, and ensure that all routing weights remain non-negative.

To enhance system resilience, OpenAI has implemented several protection mechanisms, including penalties that automatically reduce routing weights for outlier engines to facilitate their recovery or maintenance.

Retry caps dynamically prevent 'retry storms' by adjusting retry allowances based on system utilization, while proactive load shedding intentionally drops a portion of traffic as a last resort to prevent total system failure and ensure graceful degradation.

Production systems frequently encounter instabilities, with clusters and GPUs prone to failure or degradation due to issues like mysterious networking problems.

The primary goal is to maintain the health and operational status of the production system even under heavy traffic loads, utilizing proactive strategies such as penalties, retries, and load shedding.

This mechanism provides valuable time for either self-recovery or for human intervention, such as maintenance or hardware replacement, to occur without immediate system collapse.

Retries, while often necessary, can exacerbate system instability under high utilization by creating a 'retry storm' where increased load leads to more failures and subsequent retries.

To combat this, retry caps are implemented dynamically, allowing for more retries during normal operations but significantly fewer during periods of heavy load to prevent cascading failures.

Load shedding serves as the ultimate defense mechanism when system capacity is insufficient to meet demand.

This involves proactively rejecting a portion of incoming traffic to prevent total system failure, ensuring that the system degrades gracefully rather than experiencing a complete outage.