How do you set up a Kubernetes cluster for AI workloads in 2027?

What an AI-ready cluster actually requires

A general-purpose Kubernetes cluster and an AI cluster share a control plane and almost nothing else. The default scheduler was designed for stateless web services: small, fungible pods that can start one at a time, tolerate eviction, and be rescheduled anywhere. AI workloads invert every one of those assumptions, and the setup work is mostly about closing that gap.

The first difference is indivisibility. A distributed training job on eight nodes with eight GPUs each is not sixty-four independent pods — it is one job that either has all sixty-four ranks running or is doing nothing while burning the allocation. The default scheduler will happily place forty of them, leave twenty-four pending because another job grabbed the capacity, and produce a deadlock where two half-scheduled jobs each wait for resources the other holds. This is the single most common failure in a naive setup, and it is why gang scheduling is not optional.

The second is hardware opacity. Kubernetes natively understands CPU as a divisible millicore quantity and memory as bytes. A GPU is neither. It arrives as an extended resource — nvidia.com/gpu: 8 — that is integer-only, non-overcommittable, and invisible to the scheduler in every dimension that matters: which GPU model it is, how much VRAM it has, whether it sits behind the same NVLink domain as its neighbors, whether MIG partitioning is enabled. All of that has to be surfaced through node labels and then consumed through node selectors and affinity rules that you write yourself.

The third is the network. Standard pod networking through a CNI overlay adds encapsulation overhead that nobody notices on a REST API and that visibly slows a multi-node training run. Collective operations like all-reduce move the full gradient tensor across every rank on every step. At that volume, the difference between a 25 Gbps overlay and RDMA over converged Ethernet or InfiniBand is the difference between eighty percent scaling efficiency and forty. Multi-NIC support through Multus and the RDMA device plugin belongs in the initial build, not a later migration, because retrofitting it means recreating node pools.

The fourth is data gravity. A training job reading from object storage through a naive fuse mount will spend a large fraction of wall-clock time waiting on I/O while expensive accelerators sit idle. GPU utilization dashboards that show sixty percent when the job is supposedly compute-bound are almost always describing a storage problem, not a model problem.

The fifth is the container itself. AI images are large — a CUDA base layer plus PyTorch plus dependencies routinely lands in the multi-gigabyte range, and images carrying model weights go higher. Pulling that image cold onto a new node before a pod can start adds minutes to every scale-up. Registry placement, image pre-pull, and layer caching are startup-latency features, not conveniences.

Everything in the sections below is a response to one of these five pressures. If a decision does not reduce scheduling deadlock, hardware opacity, network overhead, I/O stall, or cold-start latency, it is probably not load-bearing for an AI cluster.

The step-by-step process

Work in this order. Each stage assumes the previous one is verified, because debugging a distributed training failure on top of an unverified driver stack wastes days.

Stage one: control plane. Use a managed control plane unless you have a specific reason not to. The major clouds all charge roughly a tenth of a dollar per hour per cluster for control plane management, which is rounding error next to a single GPU node and buys you a maintained, HA etcd you do not have to operate. Self-managed control planes make sense on-premises or when you need custom admission webhooks in the API server's own configuration, and they cost roughly one dedicated SRE's attention. Pin a Kubernetes version that is one or two minor releases behind the newest — GPU operators and device plugins trail upstream by weeks to months, and being on the bleeding edge means discovering that your driver DaemonSet does not support the new kubelet resource API.

Stage two: node pools, separated by role. Create at least three. A CPU system pool of three to five small nodes runs the operators, controllers, monitoring stack, and ingress — never let these compete with training pods. A GPU training pool holds the large multi-GPU instances and should be tainted so only workloads that explicitly tolerate the taint land there. A GPU inference pool holds smaller, cheaper accelerators for serving. Keeping training and inference in separate pools matters because their scaling behavior is opposite: training pools want to scale to zero between jobs, inference pools want a warm floor that never drops below the p99 traffic level.

Stage three: the GPU stack. Install the NVIDIA GPU Operator via Helm. It manages the driver DaemonSet, the container toolkit, the device plugin that advertises nvidia.com/gpu to the kubelet, node feature discovery for labeling, and DCGM for telemetry — five separate installs that used to be manual. On managed node images that ship drivers pre-installed, disable the operator's driver component and let it manage only the toolkit and plugin; running two driver installs against one kernel produces a node that appears healthy and fails every CUDA call. Verify with a pod that runs nvidia-smi and a second that runs an actual small CUDA workload — nvidia-smi succeeding proves the device is visible, not that the runtime is wired correctly.

Stage four: batch scheduling. Install Kueue or Volcano. Kueue is the CNCF-aligned option and works by admitting Jobs against ClusterQueues with configurable borrowing between cohorts; it delegates actual pod placement to the default scheduler and is the lower-friction choice for teams already using the Job API. Volcano replaces the scheduler outright and offers richer gang scheduling, queue-level fair-share, and topology-aware placement — more capable, more to operate. Either way, define queues per team with nominal quotas that sum to less than physical capacity, and enable borrowing so idle quota is reclaimable.

Stage five: the training operator. Install Kubeflow Training Operator or equivalent so that PyTorchJob, TFJob, and MPIJob are native resources. These CRDs handle the rank-zero coordination, environment variable injection for MASTER_ADDR and WORLD_SIZE, and restart semantics that you would otherwise write by hand in every job spec.

Stage six: storage. Attach a shared filesystem for datasets and checkpoints. A parallel or NFS-backed filesystem with adequate throughput per GPU is the baseline; object storage with a caching layer in front is the common alternative when datasets are cold and large. Checkpoint writes are bursty and large — size the write path for the checkpoint burst, not the average read.

Stage seven: observability and policy. Wire DCGM metrics into Prometheus, add per-namespace ResourceQuotas on nvidia.com/gpu, set LimitRanges so a forgotten pod cannot request an entire node, and add a PriorityClass hierarchy so production inference preempts exploratory training.

Costs, timelines, and typical ranges

Cost in an AI cluster is dominated by accelerator-hours to a degree that makes every other line item nearly irrelevant. The managed control plane fee, the load balancer, the monitoring stack, and the CPU system pool together typically land in the low hundreds of dollars a month. A single node of current-generation high-end accelerators can exceed that in a day. This asymmetry should drive every optimization decision: an engineering week spent raising cluster-wide GPU utilization from forty percent to seventy percent pays for itself immediately, while a week spent shaving the system pool is wasted.

On-demand accelerator pricing across major clouds spans a wide range depending on generation and region — older inference-class cards sit at the low end, current flagship training instances at the high end, and the multiple between them is large. Reserved or committed-use discounts typically save meaningful fractions off on-demand for one-to-three-year commitments, and spot or preemptible capacity is cheaper still but comes with interruption. Spot works well for checkpointed training with frequent saves and short restart windows; it works badly for long single-shard runs and for latency-sensitive inference. The practical pattern is a committed baseline sized to steady-state demand, on-demand for burst, and spot for a clearly separated class of interruption-tolerant jobs.

For timelines, a realistic build from empty account to first successful multi-node distributed training run is two to four weeks for a team that has run Kubernetes before. Roughly: two to three days for control plane, networking, and node pools; two to three days for the GPU operator and driver verification; three to five days for scheduler, quotas, and the training operator; a week for storage throughput tuning and the first real workload, because this is where surprises live. Teams new to Kubernetes should roughly double that. Teams retrofitting an existing production cluster should assume longer, because the existing CNI, node image, and network policy decisions will each need revisiting.

The metric that governs cost is GPU utilization, and it should be measured two ways. Allocation utilization is the fraction of GPUs assigned to pods; SM utilization from DCGM is the fraction of time the silicon is actually computing. A cluster with ninety percent allocation and thirty percent SM utilization is paying full price for idle hardware, and the cause is almost always data loading, checkpoint stalls, or a job that reserved eight GPUs to use one. Untuned clusters routinely sit in the thirty-to-fifty percent SM range; well-tuned multi-tenant clusters with backfill and preemption reach the seventy-to-eighty range. That gap is the real budget.

Storage cost is the second line item worth watching. High-performance parallel filesystems price per provisioned throughput as well as capacity, and it is easy to provision for peak checkpoint bursts and pay for that ceiling permanently. Object storage with a cache tier is usually cheaper for read-heavy dataset access; a provisioned filesystem is usually better for checkpoint-heavy workloads. Egress between regions or out to the internet is the classic surprise — keep the registry, the storage, and the nodes in the same region and preferably the same zone as the training job.

Autoscaling is the other lever. Cluster Autoscaler or Karpenter scaling a GPU pool to zero between jobs eliminates idle spend entirely, at the cost of node provisioning latency on the next job — typically several minutes for the instance plus image pull. Scale-down delay should be tuned to your job arrival pattern: aggressive scale-down saves money on sparse workloads and thrashes on bursty ones. Pre-pulling large images onto nodes at boot, or using a registry mirror in-region, cuts the cold-start penalty substantially and makes aggressive scale-to-zero viable.

Where teams get it wrong

Skipping gang scheduling until it hurts. The failure is not obvious at low utilization — with plenty of spare capacity, the default scheduler places everything and nobody notices. It appears the first busy week, as partially-scheduled jobs holding GPUs they cannot use while waiting on peers that will never arrive. By then several teams have workflows built on the default behavior and the migration is disruptive. Install the batch scheduler on day one, even if the queues start permissive.

Treating GPUs as interchangeable. A job requesting nvidia.com/gpu: 8 will be satisfied by any eight GPUs the scheduler can find, including eight across four nodes with no fast interconnect between them. The job runs and produces correct results at a fraction of expected throughput, which reads as a model problem to the data scientist and never gets escalated. Fix it with node labels for GPU model, interconnect topology, and VRAM, plus pod affinity rules that force co-location within a single NVLink domain for tightly-coupled training.

Building the data path last. Storage is treated as an afterthought because it works fine on a single node with a local dataset. At scale, every rank hammering the same shared filesystem produces a thundering herd that neither the filesystem nor the network was sized for. The symptom is high GPU allocation with low SM utilization and long, uneven step times. Instrument I/O wait alongside GPU metrics from the beginning so this is visible rather than inferred.

Overprovisioning per-pod requests as a hedge. Users learn that requesting more GPUs makes their job faster or more likely to get scheduled favorably, and requests inflate until the cluster is fully allocated at low actual utilization. LimitRanges, quota enforcement, and — critically — showback that reports actual SM utilization per namespace are the corrective. People right-size when their own numbers are visible.

Version skew across the GPU stack. Driver version, container toolkit, CUDA version in the image, and the framework build inside it form a compatibility chain, and a mismatch anywhere produces errors that look like application bugs. Pin every layer explicitly, test upgrades on a canary node pool, and never let a node image auto-update in a GPU pool.

No preemption hierarchy. Without PriorityClasses, an exploratory notebook and a production inference deployment have identical claim on capacity. Define at least three tiers — production serving, scheduled production training, and best-effort exploration — with the lowest tier preemptible so it can backfill idle capacity aggressively without blocking anything that matters.

Ignoring node failure semantics. GPU nodes fail more often than CPU nodes — thermal events, ECC errors, driver hangs — and a single node loss kills an entire gang-scheduled job. Enable automated health checks that cordon and drain unhealthy GPU nodes, and require frequent checkpointing so a lost job restarts from minutes ago rather than hours.

Leaving the cluster wide open on egress and secrets. Training jobs pull from external registries and datasets and often carry credentials to object storage. Default-deny NetworkPolicies, workload identity instead of long-lived keys, and a private registry mirror should be part of the initial build.

Decision framework: when to choose what

The choices that actually matter are few, and each has a clear discriminator.

Managed versus self-managed control plane. Default to managed. Choose self-managed only when running on-premises hardware you already own, when regulatory constraints prohibit a cloud-operated API server, or when you need API server flags the provider does not expose. The operational cost of self-managed etcd on a cluster with expensive nodes is rarely worth the flexibility.

Kueue versus Volcano. Choose Kueue when your workloads are already expressed as Jobs or Kubeflow CRDs, when you want to keep the default scheduler, and when quota management between teams is the primary need. Choose Volcano when you need genuine topology-aware placement, complex fair-share policies, or gang scheduling semantics richer than admission-time all-or-nothing. Running both is possible but adds a class of bug where two schedulers disagree about a pod; avoid it.

Time-slicing versus MIG versus exclusive GPUs. Exclusive whole-GPU assignment is correct for training and for any latency-sensitive inference. MIG partitioning gives hardware-isolated fractional GPUs on supported cards and suits multi-tenant inference with predictable, modest memory needs. Time-slicing oversubscribes a GPU across pods with no memory isolation and no performance guarantee — acceptable for development notebooks and interactive experimentation, never for anything with an SLO.

Spot versus on-demand versus committed. Committed capacity for the steady-state floor, on-demand for burst above it, spot for a clearly-labeled class of interruption-tolerant work with checkpointing at intervals short enough that a preemption costs minutes. Do not put a single long training run on spot without checkpoint-and-resume already proven to work.

Single multi-tenant cluster versus per-team clusters. One cluster with namespace isolation and quotas gives dramatically better utilization through borrowing and backfill, and is the right default under roughly a few dozen teams. Split into separate clusters when compliance requires hard isolation, when blast radius from a control plane incident is unacceptable, or when teams need genuinely incompatible Kubernetes versions or CNI configurations.

Overlay networking versus RDMA. Standard CNI is fine for inference and for single-node training. The moment multi-node distributed training is on the roadmap, provision RDMA-capable instances and configure Multus plus the RDMA device plugin from the start — retrofitting requires new node pools and a workload migration.

Related questions

Do I need a separate cluster for inference and training?

Not usually. Separate node pools inside one cluster give isolation through taints and PriorityClasses while preserving utilization gains from shared quota borrowing. Split into separate clusters only for compliance-mandated isolation or when a control plane incident affecting serving is unacceptable.

How many nodes should the first GPU cluster have?

Start with three CPU system nodes and two GPU nodes. Two proves multi-node networking, gang scheduling, and collective communication actually work — the failures a single node hides. Scale from there once the smoke test passes.

Can I run this on-premises instead of a cloud?

Yes, and the Kubernetes layer is nearly identical. The differences are that you own the control plane, the network fabric, and node lifecycle, and you lose elastic scale-to-zero. On-premises economics favor sustained high utilization; cloud favors bursty or uncertain demand.

What is the single most important metric to watch?

DCGM SM utilization aggregated per namespace, alongside allocation percentage. The gap between the two is wasted spend, and it points directly at data loading, checkpoint stalls, or oversized requests.

FAQ

Do I need the NVIDIA GPU Operator if my node image already has drivers?

You still want it for the container toolkit, device plugin, node feature discovery, and DCGM telemetry — but disable its driver component. Running the operator's driver DaemonSet against a node image that already ships drivers produces a node that reports healthy while every CUDA call fails. Set the driver component to disabled explicitly in the Helm values rather than assuming detection handles it.

Why do my multi-node jobs run slower than the same GPU count on one node?

Almost always interconnect. Eight GPUs on one node communicate over NVLink at very high bandwidth; eight GPUs across four nodes communicate over whatever the pod network provides. Without RDMA and topology-aware placement, collective operations dominate step time. Check whether the scheduler co-located ranks and whether RDMA devices are actually attached to the pods rather than just present on the nodes.

Should I use time-slicing to fit more users on fewer GPUs?

For development notebooks and interactive exploration, yes — it materially improves access without buying hardware. For anything with a latency target, no. Time-slicing provides no memory isolation, so one tenant can exhaust VRAM and crash the others, and it provides no performance guarantee. Use MIG where hardware isolation is needed and exclusive assignment where performance is.

How do I stop one team from consuming the whole cluster?

ResourceQuota on nvidia.com/gpu per namespace sets the hard ceiling, LimitRange prevents a single oversized pod, and the batch scheduler's ClusterQueue with a nominal quota plus controlled borrowing governs fairness over time. Add showback reporting actual utilization per namespace — visibility changes behavior faster than enforcement does.

Is scale-to-zero on GPU pools worth the cold-start penalty?

For training pools with sparse or scheduled job arrival, clearly yes — idle accelerators are the largest avoidable cost in the cluster. Reduce the penalty by mirroring the registry in-region and pre-pulling base images at node boot. For inference pools serving live traffic, no: keep a warm floor sized to p99 demand and scale above it.

What Kubernetes version should I target?

One or two minor versions behind the newest release your provider offers. GPU operators, device plugins, and batch schedulers trail upstream, and running the newest version means being the person who discovers an incompatibility. Pin the version explicitly, disable node image auto-upgrade on GPU pools, and test upgrades on a canary pool before touching the training fleet.

Sources

• https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
• https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html
• https://kueue.sigs.k8s.io/docs/overview/
• https://volcano.sh/en/docs/
• https://www.kubeflow.org/docs/components/training/
• https://github.com/NVIDIA/k8s-device-plugin
• https://github.com/k8snetworkplumbingwg/multus-cni
• https://karpenter.sh/docs/
• https://docs.nvidia.com/datacenter/dcgm/latest/index.html
• https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/

Related on PULSE

• How do you keep GPU utilization high in a shared machine learning cluster?
• What does it cost to run a self-hosted LLM inference stack?
• When should a team move AI workloads from cloud to on-premises hardware?
• How do you set resource quotas and chargeback across engineering teams?
• What belongs in an MLOps platform versus an application team's own code?