How do you handle GPU scheduling on Kubernetes for AI workloads in 2027?

What GPU scheduling on Kubernetes actually is, and why default behavior fails you

Kubernetes was designed around CPU and memory — two resources that are compressible, divisible, and forgiving. You can ask for 250 millicores and get throttled rather than killed. GPUs break every one of those assumptions. A GPU is exposed to Kubernetes as an *extended resource* (nvidia.com/gpu), which means it is an integer-only, non-overcommittable, all-or-nothing count. You cannot request nvidia.com/gpu: 0.5 in a stock cluster. You request 1, or 2, or 8, and the scheduler either finds a node with that many free or your pod sits Pending forever.

That integer-only model is the root of most GPU waste in AI infrastructure. A Jupyter notebook doing exploratory data analysis on a 1GB dataset holds an entire 80GB H100 hostage. An inference pod serving a 7B parameter model at 40 requests per second might use 18GB of VRAM and 25% of the streaming multiprocessors, but Kubernetes has reserved the whole card for it. Teams routinely discover, once they install DCGM monitoring, that their fleet-wide GPU utilization sits somewhere in the teens or twenties while engineers complain there are no GPUs available. Both things are true simultaneously — the GPUs are *allocated*, they are just not *busy*.

The mechanics underneath are worth understanding because they explain what you can and cannot fix. GPUs reach pods through the Kubernetes device plugin API. The NVIDIA device plugin runs as a DaemonSet, discovers the physical devices on each node, and advertises them to the kubelet, which republishes them as allocatable extended resources on the Node object. When a pod requesting nvidia.com/gpu: 1 is bound to that node, the kubelet asks the plugin for an allocation, and the plugin returns the environment variables and device mounts that the NVIDIA container runtime uses to inject /dev/nvidia* character devices into the container. The scheduler itself is nearly ignorant of what a GPU is. It counts integers.

That ignorance is the second problem: the default scheduler schedules pods one at a time. For a web service that's fine. For a distributed training job across 8 pods that must all rendezvous through NCCL before step one, it's catastrophic. If the cluster has 6 free GPUs, the default scheduler will happily place 6 of your 8 pods, which will sit in a rendezvous barrier holding 6 GPUs hostage, while the other 2 wait for capacity that will never arrive because your own job is consuming it. Two such jobs deadlock each other indefinitely. This failure mode has a name — resource deadlock — and the fix is gang scheduling, sometimes called all-or-nothing scheduling: place all N pods or place none.

The third gap is fairness. Stock Kubernetes has no concept of a team quota that borrows idle capacity and gives it back under pressure. Namespace ResourceQuota gives you a hard ceiling, which is worse than nothing on expensive hardware — if the research team's quota is 16 GPUs and they're using 4, the other 12 are locked away from everyone else even when the cluster is starved. What AI platforms actually need is *elastic* quota: a guaranteed floor per team, the ability to borrow above it when the cluster is idle, and reclaim with preemption when the owning team comes back. That is exactly what Kueue's ClusterQueue cohorts and Volcano's queue weights implement.

The adjacent lesson, and one worth carrying into related infrastructure work: the same three gaps show up whenever you schedule any scarce, indivisible, expensive accelerator — TPUs, Habana Gaudi HPUs (habana.ai/hpu), AMD Instinct cards (amd.com/gpu), FPGAs, even high-memory nodes for feature engineering. The device plugin pattern is identical, and so are the batch-scheduler fixes. If you build the topology-aware, gang-scheduled, quota-managed layer once, it transfers.

The step-by-step process for standing up GPU scheduling

Work bottom-up. Each layer depends on the one below it, and skipping a layer produces symptoms that look like bugs in the layer above.

Step 1 — Get drivers and runtime on the nodes. Install the NVIDIA GPU Operator via Helm. It deploys, in dependency order, the driver container (or validates a pre-installed driver), the NVIDIA container toolkit, the device plugin, DCGM and DCGM-exporter for metrics, node-feature-discovery for GPU labeling, and the MIG manager. Doing this by hand is possible and miserable; the operator exists because driver/toolkit/kernel version drift across a heterogeneous fleet is a genuine time sink. Validate with kubectl get nodes -o json | jq '.items[].status.allocatable."nvidia.com/gpu"' — if that returns nulls, nothing above this layer will work.

Step 2 — Label your fleet honestly. Node-feature-discovery will tag nodes with GPU product (nvidia.com/gpu.product=NVIDIA-A100-SXM4-80GB), memory, compute capability, and MIG configuration. Add your own labels for interconnect topology — whether nodes have NVLink/NVSwitch internally and InfiniBand or EFA between them. Distributed training performance is dominated by interconnect, and a scheduler that spreads an 8-pod job across 8 separate PCIe-only nodes will produce a job that runs at a fraction of the speed of the same job packed onto one NVSwitch node. Encode that reality in labels now so the scheduler can honor it later.

Step 3 — Choose your sharing mode per node pool, not per cluster. You have three real options for splitting a card. MIG (Multi-Instance GPU) is hardware partitioning available on A100, H100, and later data-center cards: it carves a physical GPU into up to seven isolated instances with dedicated SM slices, memory, and memory bandwidth. MIG gives true fault and performance isolation — one instance cannot OOM or slow down another. Time-slicing is the device plugin's software option: you declare a replication factor and the plugin advertises, say, 4 virtual GPUs per physical GPU, with the driver context-switching between them. There is no memory isolation whatsoever — one greedy pod OOMs everybody on that card. MPS (Multi-Process Service) sits in between, allowing concurrent kernel execution from multiple processes with optional memory limits, better throughput than naive time-slicing, still weaker isolation than MIG. The practical pattern: a MIG-partitioned pool for inference and notebooks, a whole-GPU pool for training, and node selectors or taints keeping the two workloads apart.

Step 4 — Install a batch scheduler. For gang scheduling with HPC-style queues, use Volcano, a CNCF project that ships its own scheduler binary and PodGroup/Queue CRDs. For a more Kubernetes-native, quota-first model that works with upstream Job, JobSet, Kubeflow training operators, RayJob, and MPIJob, use Kueue, maintained by SIG Scheduling, which suspends Jobs until admission and then unsuspends them once quota is reserved. Kueue's ResourceFlavor abstraction is the piece people underuse — it lets you model "A100 on-demand," "A100 spot," and "H100 on-demand" as distinct flavors with separate quotas and an ordered preference list, so jobs fall back gracefully.

Step 5 — Model your quotas as a cohort. Create one ClusterQueue per team with a nominal quota that reflects their guaranteed floor, put all of them in the same cohort, and set borrowingLimit and lendingLimit. Idle capacity flows to whoever needs it; when the owning team submits, preemption reclaims it. Set reclaimWithinCohort: Any and withinClusterQueue: LowerPriority to get both cross-team reclaim and in-team priority ordering.

Step 6 — Make preemption survivable. Preempting a training job that has run for nine hours and never checkpointed is destroying nine hours of GPU time. Before you enable preemption, require checkpointing: every N steps to persistent storage, plus a preStop hook and a terminationGracePeriodSeconds long enough to flush (60–300 seconds is typical for large models). Frameworks make this easy — PyTorch Lightning, DeepSpeed, and the Kubeflow training operators all have checkpoint/resume paths. Without this discipline, preemption converts a utilization win into a throughput loss.

Step 7 — Autoscale the node layer. Cluster Autoscaler or Karpenter provisions GPU nodes on demand. Karpenter's NodePool + NodeClass model lets you express "these instance types, prefer spot, consolidate when underutilized," and it will bin-pack pending pods onto the cheapest sufficient instance. Budget real cold-start time: pulling a multi-gigabyte CUDA image, installing drivers, and passing GPU validation commonly takes several minutes, so a naive scale-from-zero design gives your users a very slow first experience. Mitigate with a warm buffer of one or two idle nodes, image pre-pulling on the AMI, or Karpenter's node-image caching.

Step 8 — Instrument before you optimize. DCGM-exporter into Prometheus, then track four things per namespace: GPU allocation (how many cards are claimed), DCGM_FI_DEV_GPU_UTIL (are the SMs busy), DCGM_FI_DEV_FB_USED (is the VRAM actually consumed), and pending-pod queue depth with wait time. The gap between allocation and utilization is your entire optimization budget, quantified.

Costs, timelines, and the ranges you should plan against

The economics are what make this worth doing at all. GPU instances are among the most expensive line items in cloud infrastructure — top-end 8-GPU training instances run into the tens of dollars per hour on-demand, and a single such node left idle for a month costs more than a senior engineer's monthly salary in some configurations. Because the exact hourly rates change constantly across regions, commitment terms, and generations, price your own fleet from the live pricing pages rather than any number you read in a blog post. What is stable is the *shape* of the economics:

Spot and preemptible capacity is dramatically cheaper than on-demand — commonly a large fraction off, though GPU spot pools are far thinner and more volatile than CPU spot pools, and popular instance types can be unavailable for stretches. Spot is excellent for checkpointed training and batch inference, actively dangerous for latency-sensitive serving without a robust on-demand fallback. Committed-use discounts and reserved capacity cut the rate substantially in exchange for one- or three-year lock-in, which only pays off if your baseline demand is genuinely durable. Older-generation cards (A10G, L4, T4 class) are a fraction of the price of current flagships and are frequently the correct answer for inference of small and mid-sized models; teams reflexively reach for the newest silicon and then run a 7B model on it at 15% utilization.

On timelines, plan roughly like this. Getting the GPU Operator installed and validated on an existing cluster is a day or two of work for someone comfortable with Helm, longer if you have driver version conflicts or a locked-down node image. Adding Kueue or Volcano and modeling your first set of queues is a week or two, most of it spent not on YAML but on the political question of who gets what floor. Enabling MIG on a pool and migrating inference workloads onto MIG profiles is another week, including the application-side work of confirming your models fit in the profile's memory. Getting checkpoint/resume reliable across your training codebase — the actual prerequisite for preemption — is the long pole, often a month or more, because it touches every team's training script. Full monitoring and chargeback dashboards, another week or two.

The return shows up as utilization. Teams starting from naive whole-GPU allocation with no queueing commonly sit in the 15–30% average utilization range. Adding fractional allocation for inference and notebooks, gang scheduling for training, and preemptive queues that let batch jobs backfill idle capacity typically moves that into the 50–70% band, sometimes higher on well-run clusters with steady batch demand. Roughly doubling utilization means roughly halving the fleet required for the same throughput — that is where the money is, and it dwarfs any licensing decision.

On licensing, the open-source path — GPU Operator, Kueue or Volcano, Prometheus/Grafana, Karpenter or Cluster Autoscaler — costs nothing in software and quite a lot in engineering attention. Commercial GPU orchestration platforms (NVIDIA's Run:ai, HPE's Determined AI, and others in that space) charge per GPU or per node annually and buy you a management UI, workspace/notebook provisioning, chargeback reporting, and vendor support. The honest calculus: if a platform costs the equivalent of a small percentage of your annual GPU spend and it reliably moves utilization up ten points, it pays for itself immediately. If you have twelve GPUs and one platform engineer who enjoys this work, the open-source stack is fine. The crossover is less about GPU count than about how many distinct teams are fighting over the same pool.

One cost people forget entirely: storage and data pipeline. A GPU stalled waiting on data is as expensive as a GPU sitting idle, and it looks like a utilization problem when it's really an I/O problem. If DCGM shows low SM utilization while your job is "running," profile the input pipeline before you buy more GPUs. Fixes range from more dataloader workers and prefetching, to local NVMe caching of the hot dataset, to GPUDirect Storage paths that bypass the CPU bounce buffer entirely. This is the single most common misdiagnosis in AI infrastructure — treating a data-loading bottleneck as a scheduling problem.

Where teams get GPU scheduling wrong

Confusing allocation with utilization. The dashboard says 100% of GPUs are allocated, so leadership concludes the cluster is saturated and approves a hardware purchase. DCGM says average SM utilization is 22%. You did not need more GPUs; you needed fractional allocation and a queue. Always report both numbers side by side, and make the gap between them a tracked metric with an owner.

Turning on time-slicing and calling it isolation. Time-slicing gives you oversubscription, not isolation. There is no memory partitioning — a pod that allocates a large tensor will OOM every other pod sharing that card, and those pods will fail with errors that look nothing like "your neighbor was greedy." Use time-slicing for dev notebooks and low-stakes inference where a crash is an annoyance. Use MIG where a crash is an incident.

Skipping gang scheduling on distributed jobs. Covered above, but worth repeating because it's the most expensive single mistake: partial placement of a multi-pod job produces a deadlock that consumes GPUs indefinitely while making zero progress, and it gets *worse* under load, exactly when you can least afford it.

Ignoring topology. Placing an 8-way data-parallel job as eight pods on eight separate nodes connected by ordinary Ethernet, when the same job could have landed on one NVSwitch-connected node, can cost you a large multiple in step time on communication-heavy models. Use pod affinity, topology-aware scheduling features in your batch scheduler, or a JobSet/PodGroup abstraction that expresses "these pods belong together, physically."

Enabling preemption before checkpointing works. You will get one furious message from a researcher whose four-day run died at hour ninety, and preemption will be disabled permanently by political force. Sequence it correctly: checkpointing first, dry-run preemption with alerts and no eviction, then real preemption.

Hard namespace quotas as the fairness mechanism. ResourceQuota on nvidia.com/gpu gives every team a ceiling and no borrowing. On a $2M fleet, stranded capacity behind a hard quota is the most expensive kind of idle. Use a cohort model with lending and borrowing instead.

One giant homogeneous node pool. Different workloads want different silicon. Interactive notebooks want cheap, small, fast-to-start instances. Inference wants MIG slices or older-generation cards. Large training wants the flagship multi-GPU nodes with high-speed interconnect. Model these as separate node pools with taints, and let the scheduler's flavor preferences route work automatically.

No idle reaper. Interactive GPU notebooks are where utilization goes to die. Someone opens a session on Tuesday, closes the laptop lid, and the pod holds a GPU until Friday. Run a controller that watches DCGM utilization per pod and terminates interactive sessions that have been below a threshold for a set window, after warning the user in Slack. Every mature GPU platform ends up building or buying this.

Letting image pull time dominate cold start. Multi-gigabyte CUDA and framework images pulled from a distant registry on every new node turn a two-minute scale-up into an eight-minute one. Bake common layers into the node image, use a regional registry mirror, or pre-pull on node bootstrap.

Not exposing queue position to users. When a job sits Pending with no explanation, users respond by submitting it four more times, which makes the queue worse. Surface position, estimated wait, and the reason for blocking — quota exhausted, no matching flavor, waiting on gang capacity. Half of GPU scheduling complaints are actually observability complaints.

Decision framework: choosing the right scheduling stack

Start from workload shape, not from vendor comparison. The questions that actually determine your architecture, in order:

Is your dominant workload training or inference? Training is bursty, tolerant of queueing, sensitive to interconnect, and benefits enormously from gang scheduling and preemptible batch queues. Inference is steady-state, latency-sensitive, intolerant of preemption, and benefits from fractional allocation and horizontal autoscaling on request metrics rather than GPU count. Clusters that serve both should physically separate them into pools; the scheduling philosophies are opposed.

Do you run multi-pod distributed jobs? If yes, gang scheduling is non-negotiable, which means Volcano, Kueue with JobSet, or a commercial platform — not the default scheduler.

Do multiple teams contend for one pool? If yes, you need elastic quota with borrowing and reclaim. Kueue's cohort model is the cleanest open-source implementation. If a single team owns the whole cluster, skip queues entirely and use priority classes plus a good idle reaper.

Does your hardware support MIG? If your fleet is A100/H100-class and your inference models fit in a MIG profile, MIG is the highest-quality fractional path. If you're on V100, T4, A10G, or L4, MIG isn't available and your options are time-slicing, MPS, or a software-partitioning platform.

How much platform engineering can you fund? This, more than anything, decides open-source versus commercial. The open stack is genuinely capable and genuinely demanding — you own upgrades, breakages, and the internal UX. A commercial platform trades money for that ownership.

Are you on managed Kubernetes or bare metal? Managed offerings (EKS, GKE, AKS) each provide GPU node pools, driver installation paths, and autoscaling integrations, and each has its own accelerator-specific tooling worth evaluating before you build. Bare metal gives you full control over topology and no per-hour markup, at the cost of owning capacity planning and hardware failure. Many serious AI teams run a bare-metal or colo baseline for steady demand and burst to cloud for spikes — a burst design that only works if your scheduling layer is portable, which is another argument for standardizing on the Kubernetes-native stack rather than a cloud-specific one.

How this extends to adjacent infrastructure problems

The patterns here generalize further than most teams realize, and the adjacent wins are often cheaper than the core ones.

Other accelerators use the identical mechanism. Habana Gaudi exposes habana.ai/hpu, AMD Instinct exposes amd.com/gpu, and cloud TPUs expose their own resource names — all through the same device plugin API, all integer-only, all subject to the same gang-scheduling and quota needs. If you build your platform around Kueue flavors rather than hardcoded GPU assumptions, adding a second accelerator type later is a configuration change rather than a rewrite. That optionality has real negotiating value when you're pricing capacity.

CI and evaluation pipelines are a scheduling problem too. Model evaluation suites, nightly regression benchmarks, and data-processing DAGs all want GPUs intermittently and are perfectly tolerant of preemption. They are the ideal backfill workload — give them the lowest priority class and let them consume whatever the training queue isn't using. Many clusters that appear fully utilized are simply missing this bottom tier.

Serving autoscaling is a different discipline than batch scheduling. For inference, the metric that matters is queue depth or p99 latency, not GPU count, so you want KEDA or a custom metrics adapter driving HPA on request-level signals. Newer inference-serving projects push further, routing requests based on KV-cache locality and model-server load rather than round-robin. If your inference pods are autoscaling on CPU utilization, you have a scheduling bug hiding in plain sight.

Cost attribution closes the loop. Once you have per-namespace GPU-hours from Prometheus, publish them. Teams behave differently when their GPU consumption is visible next to their peers'. Chargeback — even purely informational chargeback with no money moving — is one of the highest-leverage, lowest-effort utilization interventions available, and it requires no scheduler changes at all.

Multi-cluster is the next ceiling. Once one cluster is well-scheduled, the next constraint is capacity fragmentation across clusters and regions. Kueue's MultiKueue and federation approaches let a job submitted in one place run wherever capacity exists. Don't start here — a well-tuned single cluster beats a poorly-tuned federation every time — but know it's the direction the problem eventually goes.

Related questions

How do I give a pod less than one whole GPU?

Stock Kubernetes only accepts integer GPU requests. Use MIG on A100/H100-class hardware to expose hardware-isolated instances as distinct resource names, or enable time-slicing/MPS in the NVIDIA device plugin to advertise multiple replicas per card. MIG isolates; time-slicing does not.

What is gang scheduling and do I need it?

Gang scheduling places all pods of a job simultaneously or none at all. You need it for any distributed training job where workers must rendezvous. Without it, partially placed jobs hold GPUs while making no progress and can deadlock the cluster. Volcano and Kueue both provide it.

Should I run AI workloads on spot GPU instances?

Yes for checkpointed training and batch inference — the discount is substantial. No for latency-sensitive serving without on-demand fallback. GPU spot pools are thinner and more volatile than CPU pools, so design for interruption: frequent checkpoints, graceful termination handlers, and automatic requeue.

How do I find out whether my GPUs are actually busy?

Deploy DCGM-exporter with the GPU Operator and scrape it into Prometheus. Compare allocated GPU count against DCGM_FI_DEV_GPU_UTIL and framebuffer memory used, broken down by namespace. The gap between allocated and busy is your entire optimization opportunity, expressed as a number.

Do I need a commercial GPU platform or is open source enough?

Open source — GPU Operator plus Kueue or Volcano — is fully capable but demands ongoing platform engineering. Commercial platforms buy you a UI, chargeback, workspace provisioning, and support. The crossover point is driven by how many teams contend for the pool, not by raw GPU count.

FAQ

Why is my GPU pod stuck in Pending forever?

Run kubectl describe pod and read the scheduler events. The common causes are: no node has enough free nvidia.com/gpu; the device plugin isn't running so nodes advertise zero allocatable GPUs; a taint on the GPU nodes has no matching toleration on the pod; a node selector references a label that doesn't exist on any node; or a batch scheduler is holding the job because quota is exhausted or gang capacity isn't available. Check allocatable GPUs on nodes first — that one line of jq rules out half the possibilities immediately.

Can two pods share the same physical GPU safely?

Safely, only with MIG, which enforces hardware-level memory and compute isolation between instances. Time-slicing and MPS both allow sharing but provide no memory protection — one pod allocating too large a tensor will cause out-of-memory failures in its co-tenants, and the resulting errors are confusing to debug because they surface in the innocent pod. Use MIG for anything where a neighbor's crash would be an incident, and time-slicing only for development notebooks and experimentation.

Volcano or Kueue — which should I install?

Volcano is a full replacement scheduler with HPC heritage: strong gang scheduling, queue weights, backfilling, and its own PodGroup/Queue CRDs. Kueue is a SIG Scheduling project that layers on top of the default scheduler by suspending and admitting Jobs, with a cleaner quota/cohort model and first-class integration with upstream Job, JobSet, Kubeflow operators, RayJob, and MPIJob. If your priority is elastic multi-tenant quota and native ecosystem integration, start with Kueue. If your priority is HPC-style batch semantics and fine-grained scheduler control, start with Volcano.

How do I stop idle notebooks from hoarding GPUs?

Build or deploy a reaper controller that watches per-pod DCGM utilization and terminates interactive sessions below a utilization threshold for a sustained window — a few hours is a common setting. Warn the user in Slack before terminating, and make sure notebook state persists to a PVC so a reaped session is an inconvenience rather than lost work. Pair this with short default TTLs on notebook pods that users can explicitly extend.

Does preemption lose my training progress?

Only if you haven't implemented checkpointing. Configure your framework to checkpoint to persistent storage every N steps, set terminationGracePeriodSeconds generously enough to flush a final checkpoint, and add a preStop hook that triggers the save. With that in place, preemption costs you the work since the last checkpoint — usually minutes. Without it, preemption costs you the entire run, which is why you must sequence checkpointing before enabling preemption.

What single metric best tracks whether my GPU scheduling is working?

The ratio of GPU-hours actually computing to GPU-hours allocated, tracked weekly per team. Allocation alone flatters you; utilization alone hides queueing. The ratio captures both the waste from over-allocation and, when paired with queue wait time, the pain from under-provisioning. Report those two numbers together and most GPU capacity arguments resolve themselves.

Sources

• Kubernetes: Schedule GPUs
• Kubernetes Device Plugin API
• NVIDIA GPU Operator Documentation
• NVIDIA Multi-Instance GPU (MIG) User Guide
• Kueue Documentation
• Volcano Scheduler
• NVIDIA DCGM Exporter
• Karpenter Documentation
• Google Cloud: GPUs on GKE
• Amazon EKS: Machine Learning on EKS

Related on PULSE

• How do you run distributed model training on Kubernetes?
• How do you cut cloud infrastructure costs without slowing teams down?
• How do you monitor machine learning workloads in production?
• How do you decide between building and buying an ML platform?
• How do you handle autoscaling for AI inference services?