LLM Benchmarks Face Reproducibility Issues

These systems frequently struggle to maintain the required load during testing, leading to underreported throughput. Furthermore, inherent randomness within the datasets used for evaluation often results in a lack of reproducibility, making it difficult to consistently compare model performance across different tests or environments. These issues, combined with problems related to dataset quality, further complicate the generation of reliable benchmark results.

Reliability issues are highlighted by common failures, such as a benchmark harness's inability to maintain the required load or reproduce consistent results due to randomness in datasets. Such problems undermine the validity of performance metrics. The quality of the datasets themselves also contributes to the unreliability of these evaluations.

This limitation means that benchmark harnesses running as single processes are frequently unable to achieve their target Queries Per Second (QPS), often capping performance at the level of a single CPU core when CPU-bound.

For example, attempts to reach 200 QPS on smaller hardware resulted in only 38 QPS, and even larger machines were capped at 170 QPS due to these process limitations. This issue is compounded by the fact that many harnesses inaccurately report the actual achieved throughput to the user, masking the true extent of the performance bottleneck caused by the GIL.

Benchmark clients can artificially inflate perceived latency, creating a misleading impression of server-side performance issues. This occurs when the harness thrashes due to difficulties collecting streaming token requests, leading to significant delays that are not indicative of the server's actual response time.

For example, one test observed delays of up to 58 seconds, which users might mistakenly attribute to a bottleneck in their server rather than a limitation of the benchmark client. Properly scaled harnesses demonstrate that simulated servers exhibit minimal latency, underscoring that these inflated delays are often a client-side phenomenon.

One critical factor is the arbitrary setting of model temperature to zero, which can artificially inflate throughput by making model outputs more deterministic than they would be in real-world scenarios. This manipulation creates a false impression of higher performance.

Furthermore, different benchmark harnesses employ varying methods for sampling or truncating the same dataset, leading to discrepancies in input token counts between tools. This lack of transparency regarding dataset handling means users often have no insight into how their data is being processed, further compromising the comparability and validity of benchmark outcomes.

The InferenceLLM project, originating from the Kubernetes working group and now a CNCF project, addresses critical shortcomings in LLM benchmarking by introducing standardized load generation and enhanced metric fidelity. It supports user-defined declarative configurations, significantly improving reproducibility and enabling consistent testing environments. A key innovation is its new load generator, which effectively bypasses the limitations of Python's Global Interpreter Lock (GIL) through the use of multiple processes. This multi-process approach ensures that benchmarks can accurately simulate real-world loads without being constrained by single-threaded performance bottlenecks.

This robust load generator provides a higher fidelity of metric reporting by capturing both client and server metrics, allowing for a comprehensive understanding of performance that was previously obscured by client-side issues. The project aims to provide a more accurate and reliable way to evaluate LLMs, ensuring that reported throughput and latency truly reflect the model's capabilities.

By integrating client and server metrics, InferenceLLM enables a clearer distinction between actual server performance and any artificial inflation caused by the testing harness, helping developers make more informed decisions.