GPU Performance Hinges on Operator Capability, Not Just Hardware

Oliver Kennon, a research analyst at SemiAnalysis, introduced the core argument that not all GPUs are deployed equally. He stated that a GPU's effectiveness is ultimately determined by the operator, explaining that buying hardware only provides potential.

Kennon articulated that the operating capability is what truly dictates how much useful work a GPU can deliver. This leads to the central question of who can produce the required intelligence at the lowest cost, underscoring that cost is a critical factor in the compute market.

Kennon explained that in a commodity market, producers vie to deliver comparable output, and a lower production cost provides greater flexibility to compete when prices decline. He described a 'cost curve' as a framework that orders producers by the expenses they incur to deliver the same usable work.

If two producers sell at the same price, the one with lower production costs retains a larger profit margin. The ideal metric for this market is the price per successfully completed task at the required quality, though this is not yet standardized. As a practical proxy, price per token is currently used for comparable workloads and quality requirements.

GPU dollar-per-hour price is considered an input cost rather than a measure of output, with customer prices differing from the producer's actual cost of performing the work. This distinction between cost and price is crucial for understanding the market dynamics.

GPU hours are considered input, while intelligence is the output. The inputs required to generate useful intelligence begin with hardware and electricity, specifically GPUs for this presentation. Software and operations function as a multiplier on these initial inputs.

The operator is responsible for selecting and implementing the software that serves the model, ensuring requests flow smoothly through the system. While machines might be available, human intervention is necessary to make them deliver usable responses to customers. The ultimate measure of success is the quantity of useful work obtained and whether it meets the specified task's requirements.

A concrete example from SemiAnalysis's public InferenceX calculator demonstrated that on identical B200 GPUs, the serving stack — which includes software, settings, and the number of concurrent requests, all operator choices — can significantly alter output. Both estimates used B200 GPUs serving the same model with approximately 8,000 input tokens and 1,000 output tokens, targeting 100 output tokens per second per user (P50).

In a P90 scenario, one serving stack produced around 200 generated output tokens per second, while another yielded about 300 tokens per second. This substantial difference of approximately 50% more output from the same chip and target interactivity highlights the critical role of the operator in maximizing hardware efficiency.

Assuming the hourly bill remains constant for both deployments and both meet customer requirements, an increase in output directly translates to a more cost-effective service.

With roughly 1.5 times the output, the rental spend per acceptable token becomes proportionally lower. This means that the customer effectively gets more usable and useful output for the same hourly payment.

Useful intelligence extends beyond mere output. It must effectively perform the intended task, with answers being sufficient, delivered promptly, and consistently available when needed. Crucially, the information must also adhere to agreed-upon access boundaries, meaning it must be secure.

Security is a vital yet often overlooked requirement, not captured by chip performance metrics, but essential for the usability and utility of a chip. A service can be fast but rendered useless if security is compromised.

SemiAnalysis's published security research uncovered a critical vulnerability where monitoring access inadvertently exposed another customer's logs and metrics. In the cases referenced, despite the GPUs being exceptionally fast, the service experienced severe failures.

This incident underscores that ensuring cluster security requires continuous effort, similar to maintaining promised performance. The responsibility for security typically rests with the operator, highlighting its critical role in delivering reliable and useful intelligence.

An operator's ongoing responsibility involves ensuring the system functions effectively in production, which includes safely testing and deploying updates. When a faster serving update becomes available, the operator must rigorously test it against the customer's workload.

This testing involves verifying that the output continues to meet requirements, checking actual responses, and testing under load. The operator also decides the timing and method of deployment and must maintain a rollback path to revert to a previous version if any issues arise after deployment. This continuous maintenance and response to problems are crucial for system reliability.

SemiAnalysis assesses operator capability through its ClusterMAX rating system, currently in version 3.0, which evaluates managed GPU clusters. The assessment covers four key areas: provisioning, reliability, monitoring, and security. The evaluation process incorporates hands-on testing, documentation review, and customer feedback to understand how services perform not only on ideal days but also during machine failures.

It is important to note that ClusterMAX ratings specifically apply to managed clusters, differentiating them from bare metal deployments where more operational responsibility falls on the customer. While pricing and availability are considered important factors, they do not directly measure token output. The tiers are determined relative to peers, aiming to help understand who can deliver effectively.

Oliver Kennon clarified that rental price and production cost are distinct figures. For an inference producer, the GPU provider's rental bill serves as revenue for the provider but an input cost. The GPU owner covers expenses such as equipment, power, space, and support.

The inference producer's total cost includes the rental fees plus additional outlays for software, staff, and other services beyond the rental itself. If the producer owns its GPUs, the equipment investment is amortized over its working life, with running costs added. To truly understand the cost of useful intelligence, it is necessary to determine how much useful work is generated by the total expenditure, defined as the total spending over a period divided by the output customers can actually use.

Kennon stated that superior operations reduce the cost of useful intelligence. The B200 comparison demonstrated that a higher volume of acceptable output from the same paid hours distributes the inference producer's rental expense over a greater amount of useful work. For producers owning their GPUs, this principle also extends to spreading equipment investment and running costs.

However, better operations necessitate investment in aspects like testing, safe deployment, and ongoing maintenance, meaning the additional output must justify the extra spending. This advantage can be sustained if the team consistently performs this work as software and workloads evolve, but it requires continuous effort rather than being established by a single benchmark.

Your standing on the cost curve is inherently relative, depending on what competitors can deliver rather than solely on your own improvements. Kennon illustrated this with a hypothetical example: if your full cost per successful task decreases from 10 to 8, but your competitor's cost decreases from 6 to 12 in the same period, you would transition from being the cheaper producer to the more expensive one.

This dynamic allows the cheaper competitor greater leeway to reduce selling prices, assuming customers perceive both services as equally acceptable. Therefore, merely improving does not guarantee an advantage; continuous comparison of your cost per useful work against competitors is essential. This relative comparison can be overlooked in times of scarcity, where demand significantly outpaces supply, as is currently the case.

During periods of strong demand and constrained supply, producers with vastly different cost structures can still attract customers. High revenues might reduce the urgency for CEOs to focus on optimizing their position on the cost curve, often leading them to prioritize building new data centers instead.

In times of high demand and limited supply, the urgency to optimize cost differences may diminish as producers, even those with varying cost structures, can secure customers. However, every commodity throughout history has experienced fluctuations in supply and demand, and this cycle is unlikely to be different. It is crucial for businesses to maintain a historical perspective and avoid complacency during periods of scarcity.

For businesses with leverage, proactively building a cost advantage is crucial for surviving through business cycles, even if it might not seem as immediately appealing as increasing top-line revenue. Operating capability acts as a vital buffer when customers have more choices and prices are under pressure.

These advantages are most effectively developed when capital is readily available. Therefore, focusing on operating efficiency provides resilience and a lasting competitive edge during market fluctuations.

Kennon highlighted that comparable compute services require clear 'grades' to help buyers understand and distinguish between offerings. Different operational approaches applied to the same B200 chip can yield varied results, underscoring the need for a standardized descriptive framework.

A grade outlines a specific compute service and sets expectations for its buyers. This framework allows buyers to identify which services are truly comparable, although it does not imply that intelligence is interchangeable across all tasks.

Rental price benchmarks summarize the cost of GPU hours within a defined market, but they do not reflect a producer's actual production cost or the amount of intelligence delivered. Factors such as service scope, contract terms, and operating quality all influence prices without being explicitly measured by the index.

Running a rental reference, therefore, does not guarantee the amount of useful work a deployment will produce. To effectively use it as a hedge, one must understand how it tracks the actual rent paid or earned.

Compute futures contracts, which involve standardized terms for future delivery or cash settlement based on a price reference, are gaining attention. Kennon argues that rating the operator is a crucial building block for developing a credible compute futures market. Such a rating informs the quality of service, helping to define grades so buyers can identify comparable compute services.

While a rating informs this work, it does not by itself define a futures grade, establish a sound benchmark, or create sufficient trading liquidity. An effective rental price hedge helps offset changes in rental prices but does not fix failed deployments or guarantee customer payments. What matters for a hedge is whether changes in the reference track changes in one's actual rent, as unexpected discrepancies can lead to real price risks. Ultimately, for an intelligence producer, rent is just one component of the cost; the operator still needs to convert that compute into usable work.

Kennon reiterated that a GPU's performance ultimately depends on the operator. The most efficient producer is the one who can consistently meet requirements at the lowest full cost per successfully completed task. To measure this, one must count all expenditures for delivering the service and divide by the amount of output that customers can genuinely use.

Operating capability influences both the expenditure levels and the volume of useful work delivered. Competitiveness is a relative measure; even if a company improves, it risks falling behind if its competitors advance faster. Therefore, understanding the operator's impact throughout the entire cycle, not just at a single point in time, is critical, particularly as customers gain more choices and pricing becomes more competitive.