New Techniques Boost LLM Inference Speed & Efficiency

The KV cache functions as lossless memory, scaling linearly with the size of the input data. This linear scaling creates a substantial overhead, particularly for million-token sequences, which consume significant computational resources. Addressing this challenge, industry interest is increasingly focusing on advanced context and memory compression techniques.

Traditional techniques, such as Retrieval-Augmented Generation (RAG) and agent harnesses, serve as sublinear compression methods for large-scale data. These strategies condense extensive contexts, making them more manageable for models to process. This approach helps in feeding larger information sets to language models without overwhelming their memory capacity.

This method involves a fixed set of loan query vectors that cross-attend against the full KV cache. The process produces compact keys and values in a single forward pass, effectively amortizing information representation through training.

Speculative decoding enhances throughput by employing draft tokens to verify multiple token generations within a single forward pass, without compromising output quality. This fully lossless optimization technique significantly increases tokens per second. It allows for the generation of more than one token per forward pass, streamlining the inference process.

The evolution of speculation methods has progressed from using small models to more sophisticated approaches like Eagle 3. Eagle 3 generates draft tokens by training a billion-parameter model on the hidden states of the target model. This method has demonstrably outperformed earlier techniques such as Medusa and speculation based on smaller models.

'D-Flash' leverages diffusion-based language modeling to generate sequences of tokens simultaneously, moving beyond auto-regressive generation. This method can predict 8 to 16 tokens at once, a significant improvement over Eagle's one-by-one approach. Furthermore, the tokens cross-attend to each other, leading to higher acceptance rates.

This measurement was conducted with a single B200 Qwen 38B, showcasing significant gains in both token acceptance rates and tokens per second. The method utilizes an attention mask for bidirectional drafting, maintaining causal consistency while enabling bidirectional attention.

This approach combines a diffusion model with a sequential model. The primary goal is to improve token acceptance rates through the collaborative efforts of these two models, though production results are not yet available.

The performance of speculative decoding is heavily dependent on the content of prompts and responses. Continuous retraining on live system data can lead to a 20% to 2x improvement in token acceptance rates. However, this process presents challenges related to storage, data permissions, and compute overhead, and requires the speculator to be updated whenever the underlying model changes.

The future of inference engineering anticipates significant performance gains from upcoming Blackwell/Ruben systems, alongside an increasing emphasis on adopting NVFP4 data formats for local inference. A growing reliance on disaggregation and system-wide communication is expected to become increasingly important. The focus will be on efficiently moving KV cache data across diverse systems, underscoring the shift towards more integrated and distributed AI architectures.