How do you measure and improve GPU utilization?

What GPU utilization actually measures and why the number misleads

The single most expensive misunderstanding in accelerator operations is that the "GPU-Util" column in nvidia-smi reports how hard the GPU is working. It does not. NVML defines that field as the percentage of the sampling window during which at least one kernel was resident on the device. One tiny kernel occupying one streaming multiprocessor out of 108 on an A100 for the entire sampling interval reports 100%. That is a duty-cycle metric, not a throughput metric, and treating it as throughput is how teams end up paying for an eight-GPU node that delivers the arithmetic of two.

There are at least four distinct quantities people call "utilization," and separating them is the first real step in any measurement effort:

Occupancy (duty cycle). The NVML utilization.gpu figure. Answers "was the device busy?" Cheap to sample, available everywhere, nearly useless for optimization on its own. Useful as a smoke alarm — sustained readings under 40% on a training job mean something upstream is broken.

SM occupancy / achieved occupancy. The ratio of resident warps to the maximum warps the hardware can hold. Reported by Nsight Compute and by DCGM's profiling fields (DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_PROF_SM_OCCUPANCY). This tells you whether the scheduler has enough work in flight to hide memory latency. Occupancy above roughly 50% is usually adequate; chasing 100% is a well-known trap, because register-heavy kernels often run faster at 25% occupancy than at 75%.

Achieved throughput versus theoretical peak. FLOPS delivered against the datasheet number, or achieved memory bandwidth against the HBM spec. This is the metric that correlates with cost. A model training at 35% of peak BF16 FLOPS on an H100 is burning nearly two-thirds of the hourly rate on nothing. In the LLM training world this is usually normalized as Model FLOPS Utilization (MFU) — the ratio of useful model arithmetic to peak hardware arithmetic. Well-tuned large-model training runs land in a 35–55% MFU band; below 25% there is almost always a recoverable problem.

Allocation utilization. The organizational layer: what fraction of the GPU-hours you are paying for are attached to a running job at all. A cluster where every running job hits 90% SM activity but 45% of the fleet sits idle overnight has a scheduling problem, not a kernel problem, and no amount of profiling will find it.

Why this matters beyond engineering aesthetics: accelerator capacity is now a material line item, and the gap between 30% and 60% effective utilization is a direct halving of the compute cost embedded in every unit of output. For a team whose product is inference — a copilot feature, a search reranker, a document extraction service — that cost sits directly upstream of gross margin, and therefore of the revenue quality the business reports. Finance teams increasingly ask for cost-per-thousand-inferences the same way they ask for cost-per-lead. Utilization is the denominator in that ratio.

The adjacent discipline worth borrowing from is manufacturing's Overall Equipment Effectiveness: availability × performance × quality. Map it directly — availability is whether a job is scheduled on the device, performance is achieved-versus-peak throughput, quality is whether the work produced was useful (not a job that crashed at hour nine, not a hyperparameter sweep arm that was doomed from step one). A fleet at 80% availability, 45% performance, and 85% quality is running at 31% true effectiveness. That framing tends to land with executives far better than a kernel timeline does.

The measurement and improvement loop, step by step

Optimization without measurement is guessing, and the sequence matters — profiling a kernel when the real problem is a starved dataloader wastes a day. The reliable order runs coarse to fine.

Step 1 — Establish the baseline at the fleet level. Deploy dcgm-exporter into your cluster (it ships as a DaemonSet for Kubernetes and integrates with the NVIDIA GPU Operator), scrape it with Prometheus, and build a Grafana panel showing per-GPU DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_ACTIVE, DCGM_FI_DEV_FB_USED, and DCGM_FI_DEV_POWER_USAGE. The pairwise comparisons are what carry information. High GPU_UTIL with low SM_ACTIVE means a small kernel is holding the device. High framebuffer usage with low SM activity means memory is allocated but idle — often a stuck process or a leaked cache. Low power draw during a supposedly heavy job is a strong tell for stalls, since a genuinely saturated H100 pulls close to its cap.

Step 2 — Confirm at the node with nvidia-smi. Run nvidia-smi dmon -s pucvmet -d 1 for a one-second-interval stream of power, utilization, clocks, memory, and temperature. Use nvidia-smi pmon to attribute usage to individual processes when several tenants share a device. Watch for clock throttling in the pstate and clock columns — thermal or power-cap throttling looks like a software problem from the metrics layer and isn't one.

Step 3 — Timeline-profile with Nsight Systems. This is where most real wins are found. Capture a few hundred training steps and look at the CUDA HW row for gaps. Contiguous white space between kernel launches means the GPU waited. The usual culprits are host-side data loading, synchronous cudaMemcpy calls, Python-side preprocessing, and NCCL collectives blocking on the slowest rank. Nsight Systems shows CUDA API calls, kernel launches, memory transfers, NCCL communication, and CPU thread activity on one aligned timeline, so causality is visible rather than inferred. A run reporting 95% nvidia-smi utilization can show, in the timeline, that a third of wall time is cudaMemcpyAsync and dataloader wait.

Step 4 — Fix the pipeline before touching kernels. Increase dataloader workers, enable pinned memory and prefetching, move augmentation to the GPU or to a pre-baked format, use DALI or a sharded binary format instead of decoding JPEGs per step. If the timeline gaps close, stop — you are done, and you never had to write a kernel.

Step 5 — Kernel-level profiling with Nsight Compute. Only once the pipeline is clean. Nsight Compute gives per-kernel achieved occupancy, memory throughput, warp stall reasons, cache hit rates, and a roofline chart placing the kernel as compute-bound or memory-bound. That classification determines the fix: memory-bound kernels want fusion, better access coalescing, or shared-memory tiling; compute-bound kernels want lower precision, tensor-core paths, or better instruction mix.

Step 6 — Re-measure and record. Log the before/after on the same metric at the same layer. Utilization work regresses silently — a framework upgrade, a new augmentation, a changed sequence length — so the check belongs in CI or a nightly benchmark, not in someone's memory.

The toolchain, what each layer costs, and how long the work takes

The measurement stack is unusually cheap in licensing terms and unusually expensive in engineering time, which is the opposite of most observability decisions.

nvidia-smi and NVML. Bundled with the driver, no installation, no cost. Second-granularity metrics, per-process attribution, works over SSH on any node. The right tool for triage and for cron-driven sanity checks. Time to first useful reading: minutes.

DCGM and dcgm-exporter. Free to download and use; the fleet-monitoring path most clusters standardize on. Exposes device-level and profiling-level fields, integrates with Prometheus, Grafana, Kubernetes, and Slurm, and supports health checks and diagnostics that catch failing GPUs before a training run dies at hour twenty. Note that the profiling fields carry a small sampling overhead and, on some architectures, cannot be collected simultaneously with a running Nsight profile. Standing up an exporter, Prometheus scrape, and a first dashboard is typically a one- to three-day task for someone who has done it before.

Nsight Systems. Free, ships with the CUDA Toolkit, runs on Linux and Windows. Learning to read a timeline is a real skill with a genuine curve — budget a day for a first useful capture and a week before an engineer is fluent. The payoff is disproportionate: most first-time captures on an unoptimized training loop surface a fixable stall.

Nsight Compute. Free, same toolkit. Deeper curve than Nsight Systems because it assumes familiarity with the memory hierarchy, warp scheduling, and roofline reasoning. Reserve it for teams writing or tuning custom kernels, or for anyone chasing the last 20% on a workload that runs continuously.

AMD's stack. rocm-smi mirrors nvidia-smi for Instinct hardware; rocprof and the ROCm profiling tools provide hardware-counter data for compute-unit occupancy, memory bandwidth, and wavefront occupancy on HIP and OpenCL kernels. Free with ROCm. If you run heterogeneous silicon, plan on two parallel dashboards rather than one unified pane — the metric semantics do not map one-to-one.

Cloud-provider metrics. AWS CloudWatch, Google Cloud Monitoring, and Azure Monitor all surface coarse GPU utilization and memory metrics with no setup. They are adequate for capacity and billing conversations and useless for optimization — no kernel detail, no timeline. Treat them as the accounting layer, not the engineering layer.

Serving-layer metrics. NVIDIA Triton Inference Server exposes Prometheus metrics including per-model inference counts, queue and compute durations, and GPU utilization. vLLM and similar engines expose comparable metrics. For inference workloads these often matter more than device counters, because the fix — dynamic batching, concurrency limits, continuous batching — lives at the server layer.

Where the time actually goes. The realistic effort curve for a team starting cold: a week to instrument and dashboard, a week to profile the two or three workloads that consume most of the fleet, two to four weeks of iteration to land the pipeline and batching fixes. Kernel-level work is open-ended and only justified for workloads running continuously at scale.

What the improvement is typically worth. The common pattern is a training job sitting somewhere in the 40–70% duty-cycle range with a starved input pipeline; closing that gap frequently moves the job into the high 80s or low 90s. On the throughput metric the movement is usually smaller but more meaningful — a workload at 25% MFU tuned to 40% has cut its compute cost per token by roughly a third. Against a fleet costing five or six figures a month, that arithmetic funds the engineering time several times over, which is the argument that gets the work prioritized in the first place.

Where teams get it wrong

Optimizing the metric instead of the workload. Someone is told to raise GPU utilization, so they increase batch size until the duty cycle reads 98% — and step time gets worse because the larger batch spilled into slower memory paths or degraded convergence. Utilization is an instrument, not a goal. The goal is samples per second per dollar, or tokens per second per dollar. Always pair a utilization change with a throughput measurement.

Profiling the wrong layer first. Two days in Nsight Compute tuning shared-memory tiling on a kernel that accounts for 4% of step time, while the dataloader eats 30%. Always take the timeline capture before the kernel capture. Amdahl's law is unforgiving and cheap to respect.

Ignoring the multi-GPU tax. On distributed training, the slowest rank sets the pace. One node with a degraded NVLink, a thermal issue, or a slightly different driver drags every other GPU to its speed while all of them report high utilization — they are busy waiting inside a collective. Profile the whole job, compare per-rank step times, and look at NCCL timing specifically. Straggler detection belongs in your monitoring, not in a postmortem.

Confusing memory pressure with compute pressure. Framebuffer at 78 GB of 80 with SM activity at 20% is not a compute problem. Caching allocators hold memory they are not using, so a high FB_USED reading often reflects the allocator's high-water mark rather than live tensors. Read it alongside SM activity or you will make the wrong call about whether you need bigger GPUs.

Leaving small jobs on big GPUs. An inference model needing 8 GB and a fraction of the SMs, given an entire H100, wastes most of the device no matter how well the kernel is written. MIG partitions an A100 or H100 into as many as seven isolated instances with dedicated memory and compute paths; time-slicing and MPS offer softer sharing with weaker isolation. Choosing none of these is choosing to strand capacity.

Never checking clocks and power. A GPU throttling on temperature or hitting a power cap looks identical to a well-behaved job from the utilization column. Check nvidia-smi -q -d PERFORMANCE for throttle reasons before blaming the code. Also confirm persistence mode is enabled on nodes that run many short jobs — driver initialization overhead between processes is real and invisible in aggregate metrics.

Treating this as a one-time project. A framework version bump, a new augmentation, a changed sequence length, or a different attention implementation can all silently regress utilization. Without a benchmark that runs on a schedule and alerts on regression, the gains you bought with four weeks of work evaporate over the following quarter.

Measuring only during business hours. Fleet allocation utilization has a diurnal shape. If interactive notebook workloads dominate 9-to-6 and nothing backfills overnight, you are paying full price for a fraction of the day. Preemptible batch queues that soak up idle capacity are frequently the largest single utilization win available, and they require no kernel expertise at all — just a scheduler policy.

Choosing the right intervention for the symptom

Different symptoms map to different fixes, and picking the wrong one wastes weeks. The decision hinges on three questions: is the device idle or busy-but-slow, is the bottleneck upstream of the GPU or inside it, and is the workload training or serving?

Idle device, training workload. Almost always the input pipeline or a host-side synchronization. Fix ordering: more dataloader workers, pinned memory, prefetch depth, GPU-side or pre-baked augmentation, then eliminate synchronous copies and unnecessary .item() / .cpu() calls that force a device sync every step.

Idle device, serving workload. Requests are arriving one at a time and each leaves the GPU mostly empty. Enable dynamic or continuous batching at the server, set a queue-delay budget that trades a few milliseconds of latency for a large batching gain, and raise model concurrency so several instances share the device. This is usually the highest-leverage inference change available and it requires no model changes.

Busy device, low throughput, memory-bound kernels. Fuse elementwise chains, improve access coalescing, use shared-memory tiling, and adopt fused library implementations where they exist for your operator. Compilers (torch.compile, XLA, TensorRT) capture much of this automatically and are worth trying before hand-writing anything.

Busy device, low throughput, compute-bound kernels. Move to lower precision where accuracy permits — BF16 or FP8 on supported hardware — and make sure the tensor-core path is actually being taken. Check tensor dimensions against alignment requirements; unaligned shapes silently fall back to slower kernels.

Fragmented fleet, plenty of small jobs. This is a scheduling problem. Partition with MIG for hard isolation, or use time-slicing and MPS where isolation matters less than packing density. Configure the scheduler for bin-packing rather than spread, and add a preemptible queue for low-priority work.

Distributed training, all ranks busy, poor scaling. Communication-bound. Look at NCCL collective time, overlap gradient reduction with backward computation, consider gradient accumulation to reduce collective frequency, and verify the interconnect topology matches what the framework assumes.

Turning utilization work into a reportable number

Engineering wins that nobody can price get deprioritized. Translate the work into three figures a finance partner recognizes.

Effective cost per unit of output. Divide total accelerator spend by the units the platform produced — training runs completed, tokens served, documents processed. Report it monthly. A utilization improvement shows up here without any technical explanation required, and it is the number that connects infrastructure work to the margin structure sitting underneath product revenue.

Idle GPU-hours. Total provisioned GPU-hours minus GPU-hours attached to a running job. This is the scheduling metric, and it is often the largest and most embarrassing number in the deck. It also has the cheapest fixes.

Achieved-versus-peak throughput on the top three workloads. One number each, tracked over time. This is the engineering metric, and it is where profiling investment shows up.

A useful cadence: fleet dashboards reviewed weekly, a timeline profile of any workload consuming more than 10% of the fleet each quarter, and a regression benchmark on every framework or model upgrade. That rhythm keeps the gains from decaying without turning utilization into a full-time job. The adjacent lesson from capacity planning generally applies here too — the organizations that hold high utilization are not the ones with the best profilers, they are the ones that made the number visible to people who could act on it.

Related questions

Is 100% GPU utilization the goal?

No. The nvidia-smi duty-cycle figure can read 100% while the device delivers a fraction of peak throughput. Target high achieved throughput against peak — commonly 35–55% MFU for large-model training — and treat the duty-cycle number as a symptom indicator, not an objective.

Does increasing batch size always improve utilization?

Up to a point. Larger batches amortize kernel launch overhead and fill the SMs better, but past the memory ceiling you trigger spilling, recomputation, or out-of-memory errors, and very large batches can hurt convergence. Sweep batch size and measure throughput per second, not utilization percentage.

How do I tell whether the CPU is starving the GPU?

Capture a timeline with Nsight Systems and look for gaps in the CUDA hardware row aligned with busy CPU threads. High host CPU with low device activity confirms it. The fix is almost always dataloader parallelism, pinned memory, prefetching, or moving preprocessing off the critical path.

What does MIG actually change?

MIG partitions a supported GPU into multiple isolated instances, each with its own memory slice, cache, and compute paths. Small workloads that would strand most of a full device get right-sized capacity instead. The trade-off is fixed partition profiles and no dynamic resizing while instances are in use.

Can I profile and monitor at the same time?

Not always. DCGM profiling fields and Nsight captures both consume hardware performance counters, and on some architectures they conflict. Plan profiling windows deliberately — pause profiling-metric collection on the node under study, or accept gaps in the fleet dashboard during the capture.

FAQ

What is a reasonable GPU utilization target for a training job?

For the duty-cycle metric, sustained readings in the high 80s to low 90s during steady-state training are a reasonable expectation once the input pipeline is healthy. But pair it with a throughput metric — steps per second, or model FLOPS utilization — because the duty cycle alone can look excellent while the job runs well below the hardware's capability.

What is the difference between GPU utilization and SM occupancy?

GPU utilization is the fraction of time any kernel was running on the device. SM occupancy is the fraction of the hardware's warp slots that were actually filled. A kernel using one SM for the whole interval yields 100% utilization and near-zero occupancy. Read them together — the gap between them is where wasted capacity hides.

Which tool should I start with if I have never profiled a GPU?

Start with nvidia-smi dmon to confirm the symptom, then move straight to Nsight Systems for a timeline capture of a few hundred steps. That pair finds the majority of first-pass problems. Nsight Compute is a later step, appropriate once the pipeline is clean and you are tuning kernels you control.

How do I improve utilization for inference rather than training?

Batching is the lever. Enable dynamic batching in Triton, or continuous batching in an LLM serving engine, set a queue-delay budget you can afford in latency terms, and raise model concurrency so multiple instances share a device. For small models, consider MIG or MPS so several models occupy one GPU instead of each holding its own.

Do these techniques apply to AMD GPUs?

The concepts transfer directly — duty cycle versus compute-unit occupancy, roofline classification, input-pipeline stalls, batching. The tooling differs: rocm-smi replaces nvidia-smi, and ROCm's profiling tools replace the Nsight pair. Metric names and semantics do not map one-to-one, so build separate dashboards rather than trying to unify them prematurely.

How do I keep utilization from regressing after we fix it?

Add a benchmark to CI or a nightly job that runs a fixed workload and records throughput and utilization, and alert on a percentage regression. Framework upgrades, model changes, and driver updates all move these numbers. Without an automated check, the improvement decays quietly over a quarter or two.

Sources

• NVIDIA Nsight Systems Documentation
• NVIDIA Nsight Compute Documentation
• NVIDIA Data Center GPU Manager (DCGM) User Guide
• NVIDIA System Management Interface (nvidia-smi)
• NVIDIA Multi-Instance GPU User Guide
• Triton Inference Server Metrics Documentation
• dcgm-exporter on GitHub
• AMD ROCm Profiling Documentation
• CUDA C++ Best Practices Guide
• PyTorch Profiler Recipe

Related on PULSE

• [The 10 Best GPU Orchestration Tools for Kubernetes in 2027](/knowledge/ai358)
• [How do you choose an inference accelerator: GPU, TPU, or custom silicon?](/knowledge/ai415)
• [What is GPU memory fragmentation and how do you avoid it?](/knowledge/ai397)
• [The 10 Best GPU Cloud Providers for AI Training in 2027](/knowledge/ai340)
• [How do you handle GPU scheduling on Kubernetes for AI workloads?](/knowledge/ai361)
• [How do you reduce GPU costs when serving large language models?](/knowledge/ai343)