LLM Inference at Scale: Costs, Bottlenecks, and Optimizations

To maintain profitability in such a scenario, the cost per query must remain below 0.5 cents.

Inference costs are directly tied to user traffic and the number of tokens processed, increasing linearly with both factors.

Memory consumption in LLM inference significantly increases as the input context length grows.

The Time to First Token (TTFT), a critical latency metric, also slows down proportionally with larger context sizes, impacting user experience.

Traditional, vanilla implementations on local systems often process requests sequentially, which severely limits overall throughput and system efficiency, and Out-of-Memory (OOM) errors are a risk at high token volumes.

These layers encompass critical components such as normalization, attention mechanisms, and feed-forward networks.

The attention layer is identified as the most compute-intensive element within the transformer architecture, making its optimization crucial for overall efficiency.

LLM inference consists of two distinct phases: prefill and decode.

The prefill phase, which involves processing input tokens to build the KV cache and attention scores, is primarily compute-bound.

The decode phase, responsible for sequentially generating new tokens, is memory-bound due to the repeated access required for previous KV vectors, and the Time to First Token (TTFT) is largely determined by the compute-heavy prefill stage.

Because decode phases involve constant transfers of KV vectors, memory bandwidth acts as the primary constraint on speed during GPU operations.

The decode phase, which generates tokens sequentially, requires constant memory transfers for all previously computed Key and Value (KV) vectors.

This constant data movement means that the decode speed is often limited by the high bandwidth memory bandwidth, regardless of the GPU's arithmetic processing power.

The decode phase exhibits low arithmetic intensity, further highlighting its reliance on memory transfer speed rather than raw computational power.

GPU throughput is fundamentally determined by the available GPU memory capacity divided by the Key-Value (KV) cache size required per user.

Given fixed GPU and model resources, there is an inherent trade-off between accommodating longer context lengths and serving more concurrent users.

Business-specific Service Level Objectives (SLOs) for latency introduce a third critical dimension that must be considered when optimizing for performance.

For premium chat applications, the priority is typically on maintaining high quality and low latency, whereas asynchronous workloads may prioritize maximizing overall throughput.

Harshul Jain identifies that selecting an optimal GPU requires fixing a critical application requirement, such as latency or batch size, at the outset.

A capacity calculator tool helps visualize how KV size scales with increasing tokens and user counts, aiding this decision.

By focusing on the cost per million tokens rather than just hourly rates, these calculations facilitate the identification of cost-effective GPUs.

A 120 billion-parameter model requires 240GB of VRAM using BF16 precision, which far exceeds the 80GB capacity of an H100 GPU.

To address this, compression techniques such as FP8 or MXFP4 are employed, which can reduce the model's footprint to approximately 65GB.

According to Tanmay Sah, these compression strategies allow large models to fit within the physical memory constraints of available hardware.

Post-training quantization and quantization-aware training are core techniques used to manage memory efficiently without significantly degrading model quality.

Traditional attention mechanisms frequently load Query, Key, and Value matrices from High Bandwidth Memory (HBM) to compute units, leading to significant bottlenecks.

Flash Attention addresses this by dividing these large matrices into smaller tiles, allowing for faster multiplication processes directly within the GPU's faster shared memory, as explained by Tanmay Sah.

This approach efficiently tracks essential variables to calculate online softmax, minimizing redundant data transfers back to HBM and thereby enhancing processing speed.

Current LLM architectures predominantly rely on attention mechanisms for sequential token generation, but this paradigm may evolve.

Tanmay Sah suggested that future models might move towards diffusion models, which could enable simultaneous generation of all tokens, fundamentally altering current algorithms and architectural designs.

An FP16 precision model typically requires approximately 15 GB of memory.

INT8 compression effectively reduces this memory footprint to about 7.5 GB, freeing up memory for larger KV caches and allowing for either higher context limits or an increased number of concurrent users, as highlighted by Harshul Jain.

Benchmarks indicate that even higher compression ratios, such as 4x, can further reduce memory usage to 4.5 GB.

While INT8 significantly improves memory efficiency, some benchmarks paradoxically show that it may lead to a slight decrease in overall throughput.

Standard decoding processes in LLMs often recompute key and value vectors for all previously generated tokens, leading to substantial computational waste.

This redundant computation results in an O(N²) time complexity, which becomes a significant bottleneck as context length increases.

The Key-Value (KV) cache addresses this by storing these vectors in memory, allowing them to be referenced directly rather than recomputed, representing a classic trade-off where memory is utilized to reduce computational load.

In contrast, vLLM, utilizing PagedAttention and continuous batching, demonstrates an almost 15x increase in throughput, delivering significantly more tokens per second.

Further optimizations, such as prefix caching, enhance throughput and reduce the Time to First Token (TTFT), while KV quantization effectively decreases KV cache usage with minimal impact on latency.

Speculative decoding is a technique that employs a smaller, faster model to generate preliminary tokens, which are then validated by a larger 'referee' model.

Tanmay Sah expressed personal skepticism regarding the effectiveness of speculative decoding, particularly the 'Eagle algorithm'.

The primary challenge cited is the difficulty in ensuring proper alignment between the tokens proposed by the smaller model and the acceptance criteria of the larger referee model.

Benchmarking studies indicate no statistically significant difference between VLLM and SGLang when running standard, non-agentic workloads.

However, in agentic workflows that involve dynamic branching, SGLang demonstrates a substantial performance improvement, operating three to four times faster.

This performance gain is attributed to SGLang's optimizations for handling branching logic inherent in agentic environments, making it a better choice for such applications.

Despite SGLang's advantages in agentic scenarios, VLLM remains the default production solution for standard API workloads due to its robust performance in those contexts.

Researchers are increasingly prioritizing KV eviction strategies, compression, and hybrid memory architectures as the field of LLM inference evolves toward a specialized domain.

Key areas for future study include advanced KV eviction strategies, effective compression techniques for KV caches, and the development of hybrid memory architectures.

More complex challenges, such as distributed LLM inference, remain significant pain points that require further dedicated research and workshops to resolve.