Why Inference Matters More Than Training: How Providers Serve Open Models at Scale

The host opens by arguing that inference matters far more than training, because far more compute is spent serving a model than building it. He says inference investments are common at YC, and that at a previous AI company where he trained models, inference was always the bottleneck, since serving at scale without losing money on cost of goods sold is hard. He added that inference has produced over $220 billion in market cap in the last ten years.

He compares training and inference FLOPs using DeepSeek, the only company he noted that publishes such data. DeepSeek reported about 2.8 million H800 hours to train DeepSeek V3 and about 43,000 H800 hours per day on inference. He noted that this data is old and actual inference use is probably higher. Even on these numbers, inference uses about six times more FLOPs than training per year for that weight file. If the useful life of a weight file is about two years, inference FLOPs come to about 12 times training FLOPs, and that share grows as more users come online.

Serving a request has two main phases: prefill, which processes the prompt, and decode, which produces output tokens. The speaker reported per-GPU throughput of roughly 60,000 tokens per second for prefill without cache hits, about 1.5 million tokens per second with cache hits, and about 14,000 tokens per second for decode. He concluded that decode is the real bottleneck and that most chip innovation is aimed at speeding it up.

He also described how utilization is measured. Vendors quote peak figures, such as two trillion FP8 FLOPs for an H100, but the achieved share, called model FLOP utilization, is usually much lower; the speaker suggested decode utilization may be around 5%. Reported time to first token was about 400 ms at the median and about one second at the 99th percentile, with an input cache hit rate of 96%. He said that hit rate depends heavily on traffic: agent traffic caches well, while fresh chat sessions do not. One cited reason for the high hit rate is a KV cache retention policy of 72 hours.

He added that LLM inference has poor arithmetic intensity, meaning little compute per byte moved from memory, compared with diffusion models, which reuse the same weights repeatedly on a single image.

A Together speaker said open-source models can be up to ten times cheaper per year than closed models while offering nearly the same or in some cases equal or better quality. He said Together spends much of its time on how to create and serve more tokens for everyone who wants them. The company describes itself as an AI-native cloud, comparable to AWS but built around the tasks AI companies need, including pre-training, fine-tuning, inference and sandboxes for agents. It has about 80 researchers, many with PhDs, now focused on inference and GPU performance.

The speaker said the team first looks at workloads: for a coding workload, what the cache hit rate is and how long a user waits between turns, which determines how responsive the system must be.

With thousands of GPUs and requests arriving from around the world, routing combines load balancing with cache awareness. In older web services, a system would simply round-robin requests to whichever machine is least loaded. In LLM serving, sending the same request to the same GPU again gives better performance, because earlier work is already processed and stored in the KV cache.

The speaker used a coding example: the first request sends the whole codebase, and the system processes it fully. Subsequent requests can reuse that work instead of repeating it. Routing therefore has to balance load across data centers and GPUs while also choosing where the KV cache is already resident. After routing comes tokenization, which the speaker said is more involved than it looks, because requests carry tool calls such as file search or running a build command, and those must be handled correctly.

The speaker gave the example of Kimi K3, a 3-trillion-parameter model in FP4 at half a byte per parameter, which requires about 1.5 terabytes. The best B300, he said, holds only 280 GB of memory, so the model must be split across GPUs, and how to split it is a parallelism question. Prefill and decode are very different workloads, so separating them onto different GPUs, called disaggregation, can affect user experience and total throughput. Routing a request within a data center also involves many choices.

Two model-level techniques were named. Speculative decoding runs a small model alongside a large one to propose tokens; when the guess is right, those tokens come almost free. Quantization asks how many bits are needed to store each weight. The speaker said the talk would focus on kernels, parallelism and quantization. Kernels matter in the prefill and decode computation, where matrix multiplications and attention run.

Dan, speaking for Together, said expert parallelism demos have existed for two or three years, but making them reliable has only recently been solved. GB rack systems are powerful, but nodes or GPUs fail at different times. A single model replica depends on 72 GPUs, so if one of them goes out and there is no remediation or reliability layer behind it, that one failure can take out compute equal to 72 GPUs. He said Together has spent considerable time making the system resilient and able to keep running through hardware failures.

Dan explained that models are usually trained in BF16, which is 16 bits or two bytes per parameter, so a 3-trillion-parameter model takes about 6 terabytes. Machine learning computations are not exact, so for a long time it has been known that algorithms can shrink 16-bit weights to eight or four bits and still produce a correct answer. He said the difficulty is that quantization affects quality and the end-user experience, and that Together's main concern is ensuring the right quality for users.

Together tests quantized models internally before releasing them, using them itself, which he called dogfooding. The tests look at whether agent tool calls break and whether benchmark scores drop. The team also keeps a coding loop running on a Flappy Bird task; if the quantization is wrong, the bird stops flying.

One example involved MiniMax M3. Its own NVFP4 quantization lost quality, but with Together's recipe the model matched the original FP8 performance, and MiniMax could then serve more tokens. Dan presented this as a case where quantization expertise helped the people developing the core model.

Dan recalled that during his PhD he worked on SSMs and was told that nobody would use them and that transformers would dominate. Years later, those technologies appear in large models. He said Kimi K3 uses a layer called KDA, which is basically an SSM layer, described as an SSM-style linear attention layer, and that it is used in one of the best open models. Sparse attention, which nobody used for a long time, is now used, and diffusion language models are used in speculators, which he said is currently the best approach.

His students have been interested in looped models, which send tokens back through the same layers several times to increase training FLOPs without adding weights. He said he thought nobody would use the idea, but OpenAI and Anthropic were already using it. He concluded that a good architecture idea, if backed by work showing it works, can show up in GPT-6 or GPT-7 quickly.

Jeff, co-founder and CEO of Ollama, said that all open model tokens are not the same. He said the gap between the worst provider for a model and the best provider is larger than the gap between closed and open models. A 2024 paper tested 31 Llama endpoints and found that some behaved closer to GPT-3.5 than other Llama endpoints that were supposed to be identical. Jeff said OpenAI and Anthropic set an expectation of consistent quality regardless of access route, which open models do not meet, and the problem is harder to detect in advance.

He explained why this matters in agent work. If each step has a 10% chance of deviating from the intended output, a 15-step coding task could be up to 79% wrong. He said the problem gets worse as agent workloads get harder.

Reading every token is impractical, so Jeff listed monitoring methods. The most direct is running full evals. Artificial Analysis runs benchmarks several times across providers and ranks them; for Kimi K3 most providers got results right, with a few lagging. Evals catch problems such as bad quantization, but they are expensive and slow, so they cannot run continuously. Moonshot's Kimi K2 tool-call verifier scoreboard pushed providers to improve tool-call accuracy, and Jeff said Gemini later released something similar.

A cheaper option is a distribution test, comparing outputs of a reference weight with a provider's API, at roughly a dollar per test. It may miss subtle problems such as poor quantization. The cheapest option Jeff described is logprob comparison: request the next-token probability distribution and compare it with a baseline to detect drift after a new deployment or engine upgrade. A single-token request makes this cheaper still, and it can be tracked request by request over time.

Jeff said the goal is to reach an experience with open models similar to OpenAI or Anthropic. Those companies run on different hardware, such as TPUs, and he recalled a bug at Anthropic where a tiny TPU kernel bug caused what amounted to a quality outage, showing how strongly they care about serving quality. For open models, he suggested labs can share a much bigger stack for how to run their model. Typically, labs post weights on Hugging Face with example code, but that code is usually not what they serve in their own inference stack; another applied AI team often generates it.

He cited DeepSeek, which launched its own inference stack, DeepInfer, with a reported cost of about $38 per GPU hour on your own GPUs. A consistent stack lets a new inference provider have confidence in serving quality. Jeff also said many labs now quantize before launch with specialists, so the model served on their API matches what an open-model user would run, a change from earlier practice where outside parties quantized after release.

Jeff said the reality with open models is that they are the wild west: models can be customized, changed and quantized, and there are about 20 different quantizations online for almost every model, downloadable from Hugging Face or Ollama. The challenge is benchmarking them, and live customer serving needs a lighter test than full benchmarks. He called this an equally important problem locally.

He expects users to run a mix of local and cloud models. Recent local models are strong at coding, and tasks that once required cloud models may now run on a laptop, so routing will shift from choosing among cloud providers to choosing between local and cloud.

Joe, co-founder and CEO of Luminal, said his company is a compiler company, which he called unusual. He said compilers define the conversion between GPU FLOPs or GPU hours and tokens, and that the quality of the compiler largely determines whether a company loses money, breaks even, or makes a lot of money. His talk covered how Luminal uses the egg log engine, which powers its equality saturation, and how it structures its intermediate representations to target many kinds of hardware.

He said Luminal looked at neural networks and found them effectively dense linear algebra with a limited set of operations. Models are reduced to about 14 or 15 primitive operations such as add, multiply and sum. He said PyTorch has about 2,000 operations, and that the whole set can be decomposed into this simple set without losing information. He described the system as PyTorch complete rather than Turing complete: if a model can be represented in PyTorch, Luminal can ingest it and compile it to its supported backends. For example, a matrix multiply is decomposed into a broadcasted multiply followed by a sum reduction.

Luminal shares one simplified primitive IR across all its backends and wants to compile to TPUs, Trainium and GPUs. Each chip brings a set of implemented operations, which are generally low-level. On a GPU, for example, a tensor core MMA is one such operation, reading inputs from SRAM and writing outputs back. To add a backend, the team characterizes the new hardware and writes rewrite rules that equate each of its operations with operations the compiler already understands. Those equalities then propagate to the rest of the compiler.

The speaker said that this approach avoids the need for a very large compiler team and many complex rewrites. Because hardware is hierarchical, the operations may be programmed at the level of a single thread, a warp, a thread block or a full CUDA core, and mega kernels are not always the optimal solution, especially for very large models.

Once the search space is defined, the compiler needs a way to judge one program as better than another, which requires a cost model. The speaker said accurately modeling a modern GPU's cost is many PhDs' worth of work, and that a small company cannot build accurate cost models for every hardware target. Citing the bitter lesson, he said the best cost model is no cost model: run the candidate programs on the real hardware and measure how long they take. He described this as a dumb but very effective solution, and noted that some moves programmers would not think beneficial turn out to be so when measured.

The system can also prune the search. A roofline evaluation of a subprogram can show it cannot beat an already measured equivalent even in the best case, so it is discarded early. The speaker said this means far fewer profiling runs are needed than the full size of the search space would suggest.

Asked whether the search uses real inference profiles, the speaker said the team tries to use inputs as close to real ones as possible. Because Luminal runs a cloud, it knows roughly the exact tensor shapes customers will send at runtime, and it knows the data distributions. Correctness need not be checked on every run, because the rewrite rules are assumed correct unless they are wrong; by representing industry-relevant tensor shapes, the team gets fairly accurate timing. He added that the compiler can run for about 10 hours before reaching diminishing returns, and that scaling usually works well.

On debugging, he said Luminal uses PyTorch's tools that tag high-level operations and tracks that metadata through the compiler, along with the standard PyTorch profiler. Equality saturation lets the team trace a wrong output program back through the rewrites that produced it.

Asked whether the search includes quantization, speculative decoding and parallelism, Joe said generally no. The compiler handles the forward pass: a forward-pass graph is specified and compiled end-to-end within that span. It operates inside the inference engines vLLM and SGLang, which already provide speculative decoding, so Luminal piggybacks on them. The speaker said Luminal's thesis is that much of its value lies in handling very complex parallelism topologies and the complexity of the forward pass, while the rest of the innovation is left to vLLM and SGLang.

Leon Shaw, president of SGLang, said the talk covers the SGLang paper and open source work. SGLang's original idea was a radix tree inside the memory management system. Its eviction policy evicts unused leaves when memory is needed, and the scheduling and eviction algorithms can be adapted to a workload. The reason is that agents send prompts to the engine in an interleaved way. If the prompts can be reordered, they can be batched in cache-aware order and reuse context, rather than alternating between prompts A and B, which risks eviction.

He said that as GPU inference became faster while CPU efficiency did not change, a vanilla scheduling strategy could leave the GPU idle between batches: preparing one batch, running it, collecting results, then preparing the next. When the GPU was slow, bubbles of around 10% were acceptable; as GPUs got faster, the bubbles became the bottleneck, which led to overlapping the scheduling work.

Shaw said the traditional transformer does not suit long context and efficient deployment, so designs such as sliding-window attention and recurrent states like Mamba-style states have spread in open and some closed models. Recurrent states compress information into a fixed-shape state token by token. Combining these with full attention makes cache sharing hard. For a cache hit at position 10, the full attention pool must match the full KV tokens; if the sliding window is four tokens, the retained window must cover at least tokens seven through ten; and sharing the prefix from position 10 requires the recurrent state to exist exactly at position 10. If the retained tokens reach only position eight, there is no cache hit and a full attention calculation is needed for the window.

He said managing all memory pools with a single approach makes efficient sharing very hard, and that the stakes are high: raising the hit rate from 95% to 99% cuts expense about fivefold, not 5%.

Shaw described Miles, which places inference inside the training loop. The inference engine handles rollouts for RL training, and Miles supports fully asynchronous execution, so generation and training keep moving. He said Miles and the rollout backend were mainly contributed by Red Hat's team. He also reviewed the team's progression from the original paper to open source inference engines, Miles and diffusion work, and mentioned an upcoming SGLang summit with a QR code for registration.