ClusterMAX 3.0: Automated GPU Cloud Benchmarks

ClusterMAX 1.0 previously relied on manual execution of benchmark scripts for its evaluations, a labor-intensive process for thousands of GPUs.

The research team has since developed a comprehensive repository over the past 18 months, encompassing industry-standard benchmarks, custom evaluations, and specific tests for reliability and burn-in.

Testing is now highly automated, using coding agents that can autonomously run benchmarks and identify or even fix issues within the GPU clusters being evaluated.

The ClusterMAX report identified severe security issues within several cloud providers, stemming from improper implementation of network security controls.

These vulnerabilities allowed testers to gain unauthorized access to other clients' running jobs, their data, and even their storage environments.

In one alarming instance, researchers were able to view sensitive national intelligence and military-style data belonging to a particular country.

Security checks performed by ClusterMAX include verifying software and driver versions, as well as InfiniBand and Ethernet configurations, specifically looking at M-keys and P-keys.

As an accompanying resource to the ClusterMAX 3.0 report, SemiAnalysis has released a free command-line interface (CLI) tool named CMAX.

This tool enables users to verify the software versions running on their GPU clusters, helping them quickly identify potential security gaps or outdated components.

CMAX provides direct links within the terminal output, guiding users to the precise locations where software upgrades are necessary.

The new GB300 NVL72 architecture presents significant operational hurdles, requiring entirely new networking infrastructure, ARM CPUs, and Blackwell GPUs.

Mandatory direct liquid cooling is a core requirement for these systems, necessitating substantial modifications to data center facilities.

Operating system deployment and driver management for the GB300 NVL72 differ substantially from previous generations, adding to the complexity for providers.

Cloud providers that fail to adapt their infrastructure and support systems for these advanced, complex configurations are consequently seeing their rankings decline.

The report highlights that most small to mid-sized customers require cloud providers to handle advanced orchestration and manage node hot-swapping in case of failures.

While providers like Crusoe are recognized for their bare metal capabilities, they face challenges in the managed service layer, which impacts their ClusterMAX ranking.

The research aims to educate users on the practical considerations and necessary management requirements when renting AI infrastructure.

Companies such as OpenAI, Anthropic, Meta, and SpaceX are increasingly adopting a 'neocloud' strategy, essentially becoming cloud providers themselves.

These organizations are constructing their own data centers and then renting out excess compute capacity when their GPUs are not actively engaged in model training.

Many of these rental contracts include short-term cancellation clauses, often as short as 90 days, allowing the labs to reclaim their capacity as needed.

This monetization through inference and rental helps fund their core research and development initiatives.

Anthropic has significantly diversified its infrastructure strategy, expanding beyond its primary partnership with Google to collaborate with over 10 different cloud providers.

The company is increasingly leasing its own data centers, actively filling them with hardware to reduce external dependencies.

Through a collaboration with FluidStack, Anthropic is deploying TPUs directly within its self-managed facilities.

These strategic moves indicate Anthropic's transition towards developing proprietary custom chips and infrastructure, aiming to reduce its reliance on provider-specific margins and gain greater control over its compute resources.

Nvidia acquired Poolside to advance its Neotron project by integrating leading research talent into its ecosystem.

The effective cost of this acquisition is significantly reduced because the capital provided is largely reinvested by Poolside into purchasing Nvidia's own GPUs.

This strategic move creates a major new customer for Nvidia, capable of building gigawatt-scale clusters, effectively driving demand for its hardware.

The acquisition also functions as a 'defensive' strategy, preventing high-potential research teams from aligning with competitors.

The speaker suggests that Nvidia aims to prevent other companies from gaining significant influence in the ecosystem by owning key research groups.

Given Nvidia's nearly $50 billion in free cash flow per quarter, strategic investments of around 10% of this amount into such acquisitions are considered a sound business decision.

By integrating independent talent, Nvidia ensures these groups do not develop solutions that could potentially work against its hardware dominance.

Training large foundation models demands immense infrastructure, with requirements potentially scaling to 100,000 to 400,000 GPUs for frontier research.

Maintaining infrastructure reliability and conducting thorough health checks are paramount for successfully operating these large-scale model training initiatives.

Smaller companies often struggle to keep pace with the rapidly advancing open frontier in AI research, making acquisition a viable exit strategy for their talent and technology.

Globally, only approximately five companies have demonstrated the capability to build and operate GPU clusters at the 100,000-unit scale.

ClusterMAX exclusively evaluates 'managed GPU clusters' and does not assess a company's overall capabilities in data center construction or its bare metal infrastructure offerings.

For example, Crusoe was downgraded in the managed clusters ranking, yet it remains recognized as one of the world's leading data center builders.

The methodology specifically scrutinizes performance, reliability, and how providers respond to failures within their managed service offerings.

Market confusion often arises due to the blurring distinctions between managed services, inference endpoints, and bare metal offerings.

The authors of ClusterMAX categorize user feedback into requests for more detailed data and complaints regarding specific ranking positions.

For those seeking deeper weekly data, they offer a cloud Total Cost of Ownership (TCO) model.

The authors explicitly state that ClusterMAX rankings should not be interpreted as recommendations for stock investments.

SemiAnalysis maintains a separate institutional research division that conducts independent stock analysis, distinct from its technical performance rankings.

The ClusterMAX team plans to launch 'Endpoint X,' a new research initiative specifically designed to evaluate serverless inference endpoint providers.

The methodology for Endpoint X will encompass a detailed analysis of cost, reliability, security, and monitoring capabilities.

The authors acknowledge that achieving high performance in a single-batch test does not guarantee a successful partnership for scaling inference requirements.

They distinguish between market segments, noting that managed clusters, inference, and bare metal each have distinct operational requirements and considerations.

The authors openly acknowledge that they hold personal investments in some of the companies featured in their rankings, citing TensorWave and Time Intellect as examples.

They firmly assert that these investment ties do not influence the objective performance rankings published in their reports.

As evidence of their impartiality, they highlighted an instance where a company they had invested in was explicitly ranked as 'underperform.'

The ranking process is presented as an objective assessment based purely on technical performance, irrespective of any financial positions held by the authors.

Numerical correctness issues can arise when utilizing heterogeneous silicon, such as mixing B200 and B300 GPUs, due to subtle differences in their computational precision.

This increasing convergence suggests that heterogeneous silicon will likely be widely deployed across both training and inference tasks in the future.

Despite this, pre-training models continues to predominantly rely on homogeneous clusters to mitigate the risks associated with hardware inconsistencies and failures.

Nvidia is responding to this by accelerating its chip release cycle, aiming to introduce new hardware annually to maintain its market lead.

The growing hardware diversity across the industry is compelling Nvidia to innovate at an exceptionally rapid pace to prevent competitors from narrowing the performance gap and challenging its software dominance.

This rapid iteration is a direct strategic response to the increasing ease with which developers can adapt their workloads to alternative hardware platforms.