NVIDIA and OpenAI Detail AI Infrastructure Evolution

Smart GPUs are now equipped to monitor and manage their own power usage, which is critical for maximizing density within data centers. This self-management capability allows for more efficient distribution of electrical load.

Designing data center racks from the outset to handle electrical transients and sudden power spikes is essential for reliability. Incorporating intelligent power controllers and Power Distribution Unit (PDU) math schedulers enables a more dynamic and responsive power infrastructure. Ian Buck noted, "If you do that, um, you can get upwards of 30 or 40% more GPUs in the data center, but you got to design that from the beginning."

A holistic approach to designing next-generation data centers requires deep collaboration between hardware providers like NVIDIA and AI developers like OpenAI. This involves co-designing elements such as compute trays and interconnects, ensuring seamless integration and performance from the start.

Managing reliability and power effectively spans from the initial transformer all the way to the individual compute rack. A comprehensive understanding of electrical characteristics, including kilovolts (kV) and transient behaviors, is necessary to build robust and efficient AI infrastructure. Ian Buck emphasized, "Designing these next generation data centers is that you can think about the problem end to end and design in collaboration with with OpenAI in their workflows."

Upcoming data center projects are projected to reach immense capacities, potentially up to eight gigawatts, necessitating continuous co-design loops that integrate workload demands, cooling systems, and power infrastructure. This scale presents both challenges and opportunities for grid interaction.

These massive facilities can potentially act as grid stabilizers by dynamically adjusting their power consumption. Sachin Katti illustrated this, stating, "In Ohio, whenever there's spikes, we can go down on our jobs. Whenever there's gaps, we can actually go up so that we can keep the grid smooth." This adaptive load sharing can help reduce electricity costs by balancing demand during periods of high or low grid availability.

Traditional data center construction often involves designing different components in isolation, leading to inefficiencies and compatibility issues. Greenfield projects, which involve building from the ground up, offer a unique advantage by allowing for integrated design across all layers from the start.

This ground-up integration fosters a collaborative ecosystem that can modernize the energy grid itself. Sachin Katti highlighted the potential, remarking, "So today everything's designed in silos. The benefit of this greenfield project is from the ground up. We can design all of the different layers appropriately, such that this can be a very good citizen on the grid."

OpenAI is launching an AI service specifically for infrastructure optimization, demonstrating significant advancements in adapting existing code to new hardware. The Astra inference code, initially optimized for Blackwell, was successfully ported to Rubin hardware with only minor modifications required.

This porting effort yielded immediate and impressive results, achieving a three-times throughput improvement straight out of the box. Sachin Katti noted, "We literally took the same code that we had optimized for Blackwell. Run it on Rubin got 3x improvement."

The efficiency gain highlights the portability and foundational robustness of the optimized code. This success indicates that subsequent optimizations for Rubin will likely build on an already strong performance baseline.

Furthermore, the team is executing a recursive optimization loop, leveraging Astra to refine and enhance its own performance on the Rubin platform. This self-optimizing approach is expected to unlock even greater efficiencies in the future.

Engineers now submit their code to these tools before expert review, which helps to catch common issues and known bugs early in the development cycle.

These tools utilize domain-specific agents, each possessing specialized knowledge about different aspects of the software stack. Sachin Katti explained, "now we have an agenda tool that they can actually pre submit to the tool. And it looks for common issues or known bugs, because we actually have individual agents that are aware and knowledgeable about different aspects of our software stack."

Initial implementation of AI-powered code pre-checks encountered developer resistance due to slow performance, with tests taking one to two hours. This friction slowed down the development process and made tools unpopular among engineers.

By leveraging GPT's token efficiency, OpenAI significantly reduced the execution time of these checks. Sachin Katti noted, "When we first turned it on, we had like 1 to 2 our agent test, and all the developers hated it because it was like slowing the down... And by using GPT, we can do it with fewer tokens. It happens faster." Identifying 15 minutes as the 'sweet spot' for pre-check wait times proved crucial for improving the developer experience and tool adoption.

AI tools are emerging as a new and valuable layer within the existing developer toolbox, integrating seamlessly with current workflows. These tools do not aim to replace established profilers or other essential development utilities.

Instead, AI enhances productivity by complementing existing cloud-accelerated tools and enabling agents to directly utilize libraries or pre-existing tools. Ian Buck stated, "I think the it AI is another tool in our toolbox, along with all the other profilers. None of these other tools are going to be replaced. They're all just complementary."

Various software domains, such as Electronic Design Automation (EDA) and fluid simulation, currently communicate using unique domain-specific languages. This diversity presents a challenge for orchestrating AI across a wide range of applications.

With over 8,000 GPU-accelerated applications currently in use, aligning these disparate systems is critical for future productivity. Ian Buck explained that "every single domain speaks in a different language... they all have speak in a language of libraries and algorithms and concepts that are known both to the users and the agents and the AI." Standardizing these languages—encompassing libraries, algorithms, and core concepts—is essential for seamless AI integration and operation.

The industry is moving away from the past approach of trying to ignore or abstract away infrastructure complexity.

Managing this complex interplay requires tightly controlled loops to ensure optimal interactivity and performance across all system elements. Sachin Katti affirmed, "heterogeneity is a fact of life. We try to get away from not had to deal with it for so long, but I don't think we can anymore, or as we think about the future."

Small teams within OpenAI are increasingly leveraging their own internal AI models to build sophisticated system orchestration software. This strategy aims to automate and optimize complex infrastructure management tasks.

Specifically, Astra has proven to be highly effective at end-to-end optimization of these systems. Sachin Katti stated, "our push is being on how do we use our own models, how to use our own AI to build out this orchestration software... Astra is actually very good at even end to end optimization." The future strategy involves exposing telemetry data to allow these AI agents to autonomously manage and fine-tune the entire system, moving towards self-optimizing infrastructure.