Key Challenges and Solutions for LLM Inference Optimization

CTO Jin-won Lee shared the outlook that the current AI industry faces a "compute crunch," and this will continue. This is analyzed as a structural problem where the rapid development of AI models outpaces the growth of hardware computational capabilities.

While GPU compute power has increased threefold, revenue has grown tenfold, creating a significant gap. CTO Jin-won Lee explained that there are three ways to close this gap, but ultimately, only an increase in compute unit prices is a realistic solution. With inference margins currently at 70-80%, reducing the proportion of training is not practical, so an increase in inference costs is inevitable.

CTO Jin-won Lee stated that it provides 192GB of memory capacity per chip, with a target price of $5,000 (under 10 million Korean Won), enabling low-cost supply. This strategy differentiates it from existing high-cost HBM-based semiconductors, aiming for low-cost, high-efficiency AI infrastructure.

Jalapeño, unveiled at the Hot Chips conference, adopts an LPDDR-based memory structure and adjacent core computation method, showing similarities to Bertha's design philosophy. This suggests that developing AI semiconductors utilizing low-power DDR memory instead of high-cost HBM could become an industry-wide trend.

LLM inference is divided into two stages: Prefill and Decode. Prefill is a process where multiple words are input at once, and the GPU computes them in parallel, resulting in high GPU utilization. Decode, on the other hand, is an autoregressive process that outputs one word at a time sequentially, making it prone to latency. CTO Jin-won Lee explained, "Prefill is when multiple words go in at once and the GPU computes them all at once, whereas outputting character by character afterwards is the decode process, which is very sequential."

This means FFN has four times more parameters than attention. In MoE (Mixture-of-Experts) models, the proportion of FFN tends to increase even further.

In contrast, the Decode stage has a repetitive structure where one token is output for every token input. CTO Jin-won Lee explained, "In decode, one token always goes in and one token comes out, so if you don't consider batching, you can understand that data in the same form as the prefill output is circulating." These structural differences significantly impact the computational efficiency and bottlenecks of each stage.

It plays a crucial role in enabling the model to remember past conversation content and generate the next token without the user's explicit awareness. Anchor Roh Jeong-seok emphasized the importance of the KV cache, stating, "This is what we always call the KV cache."

CTO Jin-won Lee explained, "Since there isn't enough space to store KV, solutions like Mooncake discuss moving it to CPU memory or offloading it to SSD and bringing it back when needed." This is expected to enhance long-term KV cache management efficiency in an Agentic AI environment.

CTO Jin-won Lee explained, "If you calculate the point where neither computation nor memory is slower or faster, and they are balanced, it's roughly 300." If the number of operations performed with data read once is less than this point, a memory-bound phenomenon occurs, reducing computational efficiency.

This is because of the vector-matrix operation structure where one token goes in and one comes out. CTO Jin-won Lee explained, "One token goes in and one comes out, so it becomes a vector-matrix multiplication rather than a matrix-matrix multiplication, meaning the arithmetic intensity will always be very low." While attempts are made to solve this through batching, the Attention operation still faces bottlenecks due to difficulties in sharing data between batches.

CTO Jin-won Lee explained, "You can see it's about 22TB for a 512 batch. So, in fact, serving 512 users simultaneously on one or two servers is impossible, as you can see here." This demonstrates the impracticality of handling a large number of concurrent users on a single or a few servers, re-emphasizing the importance of KV cache management.

In AI services, there is a trade-off where increasing throughput inevitably means sacrificing some user responsiveness. CTO Jin-won Lee explained the relationship between throughput and interactivity using the McDonald's McMuffin order example, stating that these economic reasons also influence service pricing decisions, such as Anthropic's fast mode. This implies that a structure may emerge where faster responses require higher costs.

Traditional static batching is inefficient because the entire batch must wait until the longest request is completed. To solve this, Continuous Batching, based on the Orca paper, introduces iteration-unit scheduling. CTO Jin-won Lee explained, "Instead of per-request, let's do it per iteration. Let's reschedule every time a token is generated. Prefill and decode get mixed. This way, you can fully utilize it." This method maximizes GPU utilization by mixing prefetch and decode to fill empty slots.

To reduce memory waste during LLM inference, the PagedAttention technique has been proposed. This method applies the operating system's page table concept to memory allocation, dynamically allocating memory. CTO Jin-won Lee explained, "So, the idea to reduce that waste, which is also very simple, is to dynamically allocate memory. It brings in the page table concept. So, by dynamically allocating new pages in memory as needed, you can use exactly as much memory as you need, thus using memory efficiently and serving more users." This maximizes efficiency and allows serving more users by only using the necessary amount of memory.

In the AI industry, simply using NVIDIA GPUs and open-source models is not enough to secure competitiveness. Optimization at the infrastructure software layer is emerging as a critical differentiator and a key factor determining profitability. Anchor Roh Jeong-seok emphasized, "We might think that anyone can do business just by using NVIDIA GPUs and open-source models, but the way optimization is done within that can make a difference of several orders of magnitude in terms of money." Optimizations in orchestration layers like vLLM and SGLang can significantly increase the economic value of AI services.

Prefill-Decode Disaggregation is a concept that separates the roles of prefill and decode, based on their different computational characteristics, and assigns dedicated pools to each. CTO Jin-won Lee explained, "So, we have a prefill pool and a decode pool, and we divide them — Jensen Huang talks a lot about 'Token Factory'. You create a pool that does only one thing, according to each step." This aims to maximize effective throughput (goodput) while maintaining overall throughput and satisfying Service Level Objectives (SLOs). High-bandwidth interconnects (e.g., NVLink) are essential for this.

As long context processing becomes necessary, the importance of KV cache management grows, which can lead to new bottlenecks.

If the prediction fails, the KV cache is moved to storage.

This approach maximizes overall system efficiency through expert-unit batch size optimization.