Crusoe's Self-Healing AI Training

Manual remediation of GPU failures proves unsustainable when managing training across thousands of GPUs, according to Connor Guerrero, an engineer at Crusoe.

The Crusoe team employs a hybrid solution, combining Slurm and Kubernetes, to deliver both robustness and user-friendliness in their infrastructure-as-a-service offerings, encompassing compute, storage, and networking, alongside specialized tools for fleet orchestration.

Crusoe's platform, known as 'Managed Slurm,' is built atop Kubernetes, or CMK, integrating Slinky, an open-source project by SchedMD, for running Slurm on Kubernetes.

This architectural choice allows the system to achieve Slurm's high-performance job scheduling while leveraging Kubernetes' strengths in infrastructure resilience and observability, with 'AutoClusters' automatically detecting and replacing failed GPU nodes.

Its key features, including gang scheduling, topology awareness, and prologue/epilog functionalities for cluster validation, are familiar to researchers, who continue to prefer Slurm for its established workflows and scripts.

Dynamic modern AI workloads, involving training, post-training evaluation, and inference, frequently exceed the capabilities of traditional Slurm clusters, Young Jeong observed.

The static nature of Slurm partitions hinders adaptation to limited GPU capacity and introduces significant operational burdens, such as the manual detection and draining of unhealthy nodes.

While Slurm can identify job failures, its observability often falls short in diagnosing underlying causes like network disruptions or performance degradation.

Crusoe engineers decided to build their Managed Slurm on Kubernetes because Slurm offers essential features for AI training, such as topology-aware scheduling and familiar sbatch workflows, while Kubernetes provides a mature ecosystem for observability, networking, and self-healing.

Teams often encounter operational friction and increased costs when managing two separate infrastructure stacks for training and inference, a problem Crusoe aims to resolve.

Nikhil Gupta from Crusoe explained that their Crusoe Slurm operator coordinates with the Crusoe Managed Kubernetes platform and the broader Crusoe cloud, integrating Slurm operations directly into the existing Kubernetes control plane.

Researchers using Crusoe's platform interact with a standard Slurm interface via SSH, unaware that Kubernetes orchestrates the underlying infrastructure, while platform teams treat Slurm as another service, eliminating the need for new on-call teams or specialized runbooks.

This unified approach places all GPUs in a single cluster, enabling seamless dynamic reallocation between training and inference workloads, and allowing hardware to be shifted to training jobs during periods of low inference demand, thereby maximizing GPU utilization.

The AutoClusters system autonomously detects critical hardware issues, such as an XID 79 error rendering a GPU unusable, and immediately notifies the user without requiring manual intervention, as Nikhil Gupta outlined.

Slurm operators mark the affected node as down and signal the job to terminate, allowing users up to two minutes for checkpointing or log flushing before the system verifies replacement eligibility, then automatically cordons, drains, and replaces the unhealthy node.

When a node fails, it is automatically replaced with a healthy one drawn from spare capacity, with the system logging the remediation alert for the user.

Once the new node comes online, the Slurm job restarts, and the application code automatically loads the model and checkpoint to resume training, as described by Connor Guerrero.

Crusoe demonstrated the system's resilience by running a PyTorch training script on two A100 nodes via sbatch, then intentionally triggering an XID 79 error on one GPU to simulate a critical hardware failure.

The AutoClusters system swiftly replaced the faulty node in approximately five minutes, showcasing its autonomous remediation capabilities.

Connor Guerrero stated that the total end-to-end downtime, including application-level recovery, was less than 15 minutes, with most of that time attributed to application-level code.

Connor Guerrero introduced 'One-Click Slurm,' a utility that simultaneously provisions the Kubernetes cluster, controller, login nodes, and storage.

Engineers can then add GPU node pools with an additional single command, enabling AutoClusters by default and allowing immediate job execution.

Traditional Slurm lacks automated healing, while Kubernetes lacks high-performance job scheduling, making their integration beneficial by allowing failed GPU nodes to be automatically cordoned and signals propagated to the Slurm operator.

This integration means ML researchers can continue using familiar Slurm commands without managing containers or SSHing into pods, and GPU nodes function primarily as Kubernetes nodes, enabling inference pods to run immediately after training jobs complete.

The integrated system manages the entire remediation process, from error detection to job resumption and checkpointing, ensuring continuous operation.

Connor Guerrero explained that by synchronizing Slurm with Kubernetes, traditionally separate systems become unified, enabling automatic cordoning of failed GPU nodes and direct signal propagation to the Slurm operator.

This integration allows researchers to retain familiar Slurm commands without managing pods or containers, while platform teams maintain full access to standard telemetry and observability tools.

Connor Guerrero noted that when a Slurm job concludes, these GPU nodes can be immediately repurposed for inference pods via Kubernetes, rather than sitting idle, thereby optimizing overall platform utilization as a unified resource pool.

Because infrastructure failures are considered an inevitable aspect of the modern AI landscape, Crusoe advocates for architectural designs that autonomously manage critical errors.

Connor Guerrero emphasized that the core idea is to shift engineering focus from infrastructure maintenance to building core applications, allowing engineers to dedicate their efforts to innovation.