What Is a GPU Cluster?

Every GPU cluster splits into two node roles, plus three supporting layers: networking, storage, and orchestration software. Miss any one of them and the "cluster" is just a rack of GPUs that happen to be plugged into the same switch.

Head Nodes vs. Worker Nodes

A cluster has one head node and any number of worker nodes. The head node is the traffic controller: it schedules jobs, tracks which worker nodes are busy, and reports cluster health, but it usually does none of the actual computing. Worker nodes are where the GPUs live and where the real work happens. Each one is a server carrying GPU accelerators, CPUs for orchestration tasks the GPU shouldn't waste cycles on, high-bandwidth memory, and a network interface card fast enough to keep up with the GPUs.

Picture a construction crew instead of a single mechanic. The head node is the foreman, calling out which worker takes which task and keeping a running tally of who's free. The worker nodes are the crew doing the lifting. Neither role works without the other.

Nodes can also be homogeneous, meaning every GPU in the cluster is the identical model and memory size, or heterogeneous, meaning the cluster mixes GPU generations or vendors. Homogeneous clusters are simpler to schedule and tune. Heterogeneous clusters show up more often in organizations that bought hardware in phases and never fully replaced the old generation.

The Networking Layer

This is where clusters actually succeed or fail. If GPUs can't exchange data fast enough, adding more of them barely helps, because the cluster spends its time waiting on the network instead of computing. Inside a single node, GPUs talk over NVLink or PCIe, both fast enough that this rarely becomes a bottleneck. Between nodes it's a different problem, and this is where InfiniBand earns its reputation as the industry default for serious AI training: sub-microsecond latency and throughput up to 400 Gbps, purpose-built for exactly this job. RDMA-over-Ethernet options like RoCE compete on cost, and NVIDIA's Spectrum-X targets the same use case on a proprietary fabric.

A cluster under 8 GPUs sitting inside one server usually doesn't need any of this since NVLink or PCIe within the node is enough. The moment training spans multiple servers, inter-node networking stops being a detail and becomes the thing that decides whether the cluster performs like a cluster or like a slow single GPU with extra hardware attached.

At frontier scale the networking problem changes shape entirely. No one connects every GPU to every other GPU at full bandwidth once a cluster passes a few thousand units: the switching layers and optics required would be prohibitively expensive. Instead, operators build "islands" of GPUs with full bandwidth inside each island and oversubscribed, slower links between islands, a design SemiAnalysis documented in detail across Meta's and other hyperscalers' 100,000-GPU-class deployments (SemiAnalysis, "100,000 H100 Clusters," 2024). Reliability follows the same scaling problem in reverse. At 100,000 GPUs, even a five-year mean time to failure per network link means the first job-stopping link failure arrives in under half an hour of runtime, which is why frontier operators build hot spare nodes and fast state-reconstruction into the training loop rather than treating failure as an edge case.

Orchestration Software

Wiring GPUs together doesn't make them cooperate. That's the job of orchestration software, and two names dominate: Slurm, the workload manager that grew up in academic high-performance computing and still handles job queuing and batch scheduling better than almost anything else, and Kubernetes, the container orchestrator that scales cloud-native workloads and now runs GPU inference at most large AI shops. Ray sits alongside both, built specifically for distributed AI workloads with libraries for training, tuning, and data pipelines.

The two tools solve different problems rather than competing head-on. Slurm schedules at the job level, allocating an entire multi-node training run atomically before any task starts, which matters because a distributed training job that partially starts with some ranks running and others still waiting can deadlock or waste GPUs outright. Kubernetes takes a declarative, per-pod approach that excels at inference serving and autoscaling but needs additional layers, such as Kueue or NVIDIA's KAI Scheduler, bolted on to get the same job-level gang scheduling Slurm provides natively (Hugging Face, "Building Blocks for Foundation Model Training and Inference on AWS," 2026). That's why a meaningful share of production clusters run Slurm and Kubernetes together rather than picking one: Slurm handles the training queue, Kubernetes handles everything that looks like a service.

Training is where clusters earn their reputation. A large language model's training run starts with raw data collection, moves through preprocessing, then training itself, and that pipeline can run for weeks or months even before counting the compute needed to actually process the tokens. GPT-3 trained on 300 billion tokens and 175 billion parameters and still took months on roughly 10,000 V100-class GPUs (Brown et al., "Language Models are Few-Shot Learners," OpenAI, 2020). Newer, larger models train faster only because the clusters training them got bigger and better networked, not because the underlying math got easier: a cluster of 100,000 H100s can deliver more than 30 times the peak throughput of the 20,000-GPU cluster that trained GPT-4, though real-world Model FLOPs Utilization of around 35% at FP8 and 40% at FP16 means only a fraction of that theoretical throughput ever becomes usable training progress (SemiAnalysis, 2024).

Fine-tuning needs far less than full pretraining, typically hours or days on tens to hundreds of GPUs instead of months on thousands, but it still benefits from a cluster's parallelism, especially with production-scale datasets. Inference, running the trained model to actually answer requests, has a different profile: it's less about raw parallel throughput per job and more about serving many requests at once without users noticing a queue.

Raw GPU count is not the same thing as usable throughput. A cluster with more GPUs than it can actually keep busy is money spent on hardware sitting idle. That gap between GPUs installed and GPUs actually contributing shows up in the industry's model FLOPs utilization and model bandwidth utilization metrics, and it's often where a cluster that looks well-specified on paper underperforms in practice. The step-by-step tuning work that closes that gap, from parallelism strategy to communication libraries, is covered in our guide to optimizing GPU clusters for LLM training.

The most common mistake is treating GPU count as the only spec that matters. A 64-GPU cluster with poor interconnect can underperform a well-networked 32-GPU cluster on the same training job, because the bottleneck was never compute. It was how fast GPUs could hand data to each other. The second mistake is underestimating storage: model checkpoints, weights, and training data all have to move to every node that needs them, and a cluster with fast GPUs sitting behind slow shared storage spends its expensive hardware waiting on file reads. The third is skipping failure planning. GPU failure rates climb with cluster size, and a cluster without checkpointing loses a training run's progress the moment one card drops out mid-job, a risk that only grows as clusters push into the tens of thousands of GPUs (SemiAnalysis, 2024).