What is GPU memory fragmentation and how do you avoid it?

What it is and why it matters

A GPU allocator hands out address ranges inside a device heap. When your program asks for a 2 GB activation buffer, the allocator needs 2 GB of *contiguous* address space — not 2 GB spread across forty gaps. Fragmentation is the gap between those two things. You can watch it happen: nvidia-smi says 18 GB free on a 24 GB card, torch.cuda.memory_allocated() says you are only holding 5 GB of live tensors, and the next line of your training loop dies with an out-of-memory error asking for 1.7 GB. Nothing is leaking. The memory is simply in the wrong shape.

There are two distinct flavors, and conflating them is the root of most wasted debugging hours. External fragmentation is the classic case above: free space exists, but no single run is large enough. Internal fragmentation is the opposite waste — the allocator rounds your 513 KB request up to a 1 MB bin, and 487 KB sits inside a block that is marked "in use" and can never be handed to anyone else. External fragmentation causes hard OOM crashes. Internal fragmentation causes a slow, silent tax where reported "reserved" memory drifts far above "allocated" memory and you quietly lose headroom you paid for.

The reason this bites GPUs harder than CPUs comes down to three structural facts. First, there is no swap. A CPU process that fragments its heap leans on virtual memory and paging; a CUDA context that runs out of device memory just fails. Second, cudaMalloc is expensive and synchronizing — it can cost hundreds of microseconds and implicitly serializes streams, which is why every serious framework builds a caching allocator on top of it rather than calling it per tensor. Third, deep learning workloads are pathologically bad citizens: every step allocates and frees dozens or hundreds of intermediate activation tensors, in sizes that change whenever a sequence length, batch size, or image resolution changes.

That last point is the real engine of fragmentation. If every step allocated the exact same set of sizes, a caching allocator would reach a steady state after a handful of iterations and never fragment again — it would simply reuse the same cached blocks forever. Variable shapes break that steady state. A batch with sequence length 512 carves the heap into blocks sized for 512. The next batch at length 1,900 needs different blocks, and the 512-shaped ones are now debris sitting between live allocations. Repeat for a few thousand steps with a wide length distribution and the heap looks like Swiss cheese.

The downstream cost is not only crashes. Teams respond to OOM by shrinking batch size, which lowers throughput and GPU utilization, which raises the cost per training run or per thousand inference tokens. A fleet running at 60% effective VRAM utilization because of fragmentation needs meaningfully more accelerators than one running at 85%. On rented capacity that difference is a line item you can measure monthly. Fragmentation is therefore a capacity-planning problem wearing a systems-programming costume, and it shows up in the same budget conversation as batch scheduling, quantization, and instance-type selection.

The upstream causes are worth naming too, because the fix usually lives upstream of the allocator. Ragged input lengths, dynamic control flow, gradient checkpointing that frees and recomputes at irregular boundaries, evaluation loops that run at a different batch size than training, multi-model inference servers that load and unload weights, and long-lived processes that never restart — every one of these injects size variance into the heap. Fragmentation is what the allocator does with the variance you hand it.

The step-by-step process for diagnosing and fixing it

Work this in order. Skipping to allocator tuning before you have confirmed the failure mode is how people spend a week on environment variables to fix a genuine memory leak.

Step one: confirm it is fragmentation, not overuse or a leak. In PyTorch, print torch.cuda.memory_allocated(), torch.cuda.memory_reserved(), and the full torch.cuda.memory_summary() at the failure point. Fragmentation looks like a large gap between reserved and allocated — the allocator is holding, say, 21 GB from the driver while your live tensors total 12 GB — combined with an OOM for a request smaller than the difference. A leak looks different: allocated memory climbs monotonically across steps and never comes back down. Overuse is simplest of all — allocated is near capacity and the model genuinely does not fit. The PyTorch OOM message itself is diagnostic; it reports how much was requested, how much is free, and how much is reserved but unallocated. That last number is your fragmentation estimate.

Step two: capture a memory timeline. PyTorch ships torch.cuda.memory._record_memory_history() and _dump_snapshot(), which produce a file you can load in the browser-based memory visualizer. You get a picture of the heap over time with the allocating stack trace for every block. This is the single highest-leverage diagnostic step, and most teams never run it. What you are looking for is a few long-lived allocations parked in the middle of the address space with churn around them — those are the blocks pinning your heap into segments. Nsight Systems gives you the same story from the CUDA side if you are outside a Python framework.

Step three: kill the size variance. Before touching allocator internals, remove the variance that causes fragmentation. Bucket variable-length sequences into a small set of padded lengths — say 256/512/1024/2048 instead of every integer between 1 and 2048. Sort or group batches by length so a batch is internally homogeneous. Fix your image resolutions. Use the same batch size for evaluation as for training, or run evaluation in a separate process. Each of these collapses the number of distinct block sizes the allocator must satisfy, which is the actual mechanism by which caching allocators reach a stable steady state.

Step four: warm up deliberately. Run a few iterations at the *largest* shape you expect before the real workload starts. The allocator grabs its big blocks up front, and every subsequent smaller allocation fits inside them. Warming up at the smallest shape does the opposite: it establishes small segments that later have to be supplemented by new ones squeezed into whatever address space is left.

Step five: tune the allocator, narrowly. Only now reach for PYTORCH_CUDA_ALLOC_CONF. The two settings that matter most are max_split_size_mb, which stops the allocator from carving large cached blocks into small pieces (blocks above the threshold will not be split, so your big blocks stay big and available for big requests), and expandable_segments:True, which lets a segment grow in place using virtual memory mapping rather than requiring a fresh contiguous reservation. The second one is the closest thing to a general-purpose fix that exists today, because it attacks the contiguity requirement itself rather than the allocation pattern. TensorFlow's analogous knobs are set_memory_growth, which grows the arena incrementally instead of grabbing everything, and TF_GPU_ALLOCATOR=cuda_malloc_async, which routes through CUDA's stream-ordered pool allocator.

Step six: verify with the same instrument you diagnosed with. Re-run the memory timeline and compare reserved-minus-allocated at steady state. If the gap shrank and your peak batch size went up, you fixed something real. If reserved memory dropped but throughput dropped too, you traded fragmentation for allocator churn — empty_cache() in a hot loop is the usual culprit, because it returns blocks to the driver and forces expensive cudaMalloc calls to get them back.

Costs, timelines, and typical ranges

The honest framing is that fragmentation costs you *headroom*, and headroom converts into money at whatever your accelerator hour rate is. Put numbers on your own system rather than trusting anyone's benchmark, because the magnitude depends almost entirely on how variable your shapes are.

Measure the tax directly with one ratio: reserved memory divided by allocated memory at steady state. A well-behaved static-shape training loop settles close to 1.0 — the allocator holds barely more than what is live. A ragged-sequence workload with no bucketing can sit far above that, and every unit above 1.0 is VRAM you are paying for and cannot use. Track it as a metric alongside GPU utilization; it is cheap to emit and it tells you when a data distribution shift has quietly eaten your headroom.

On effort and timeline, the interventions sort cleanly by cost. Environment-variable tuning is minutes — you set PYTORCH_CUDA_ALLOC_CONF and relaunch, no code change, no rebuild. Warmup at max shape is a handful of lines and an afternoon including verification. Length bucketing and batch sorting is a data-pipeline change, typically a day or two, and it is the intervention with the best durable payoff because it fixes the cause rather than the symptom. Moving to fully static shapes with CUDA graph capture is a week-plus project with real constraints: graphs require fixed shapes and fixed memory addresses, so any dynamic control flow has to be flattened or captured per-shape. Custom pooled allocation through the CUDA memory pool APIs or a third-party allocator is a specialist project and rarely the right first move.

There is a runtime cost on the other side of the ledger. Padding sequences up to bucket boundaries wastes compute on padding tokens — bucket too coarsely and you can burn a meaningful fraction of your FLOPs on nothing. The tuning question is bucket granularity: too few buckets wastes compute, too many reinstates the size variance you were trying to remove. Start with four to eight buckets covering your length distribution, weighted so the buckets are narrow where your data is dense.

Similarly, expandable_segments and pool-based allocation add a small amount of virtual-memory bookkeeping per allocation. For training steps measured in tens or hundreds of milliseconds this is lost in the noise. For a latency-critical inference path serving single requests in a few milliseconds, measure it rather than assume.

Budget the diagnostic work honestly too. A first-time memory-snapshot investigation on an unfamiliar codebase realistically takes half a day: instrumenting the run, reproducing the OOM under recording, reading the timeline, and identifying the pinning allocations. That is far cheaper than the alternative most teams choose, which is halving the batch size and absorbing the throughput loss permanently.

Where teams get it wrong

Calling empty_cache() in the training loop. This is the most common mistake and it is actively harmful. torch.cuda.empty_cache() releases cached blocks back to the driver. Your next allocation then has to go through cudaMalloc again, which is slow and synchronizing, and the block you get back may be positioned worse than the one you threw away. It has a legitimate narrow use — you are handing the GPU to a different process or a different model and want the memory actually released — but as a per-step fragmentation remedy it trades a memory problem for a throughput problem and often does not even fix the memory problem.

Treating the OOM message as a request for a smaller batch size. Shrinking batch size does make the immediate error go away, and it is the right emergency action at 2 a.m. But as a permanent fix it converts a fixable systems issue into a standing efficiency loss. Worse, smaller batches often *increase* relative fragmentation, because you now do more steps per epoch and therefore more allocate-free cycles per unit of work.

Assuming nvidia-smi tells you what is happening inside the process. It reports what the driver has handed to the CUDA context, which for a framework with a caching allocator is "everything it grabbed," not "what is live." A process showing 22 GB used in nvidia-smi may hold 8 GB of live tensors. Use framework-level counters for anything about fragmentation; use nvidia-smi for who-owns-what across processes.

Sharing a GPU between processes and blaming the allocator. Two training jobs on one card each build their own cache and neither can see the other's. The failure looks like fragmentation but it is contention. Either give each job a hard fraction of the device — PyTorch exposes a per-process memory fraction cap — or partition the hardware properly so each workload gets an isolated slice. Partitioning is cleaner because the isolation is enforced below the framework.

Copy-pasting PYTORCH_CUDA_ALLOC_CONF settings from a forum post. max_split_size_mb in particular is workload-specific. Set it too low and you prevent useful splitting, inflating memory use because small requests can no longer be served from large cached blocks. Set it near your typical large-allocation size and it does what you want. There is no universally correct value, and a setting that rescued someone else's diffusion pipeline may do nothing for your transformer.

Ignoring the eval loop and the checkpoint path. Fragmentation is frequently introduced by code that runs rarely. An evaluation pass at a different batch size, a validation run with torch.no_grad() and different activation lifetimes, or a checkpoint save that briefly materializes CPU copies — each can carve new sizes into a heap that had reached a stable state. If your job survives 4,000 steps and dies right after the first eval, that is your suspect.

Chasing fragmentation when the real problem is a retained reference. Accumulating loss tensors in a Python list without calling .item() or .detach() keeps the entire autograd graph alive. The symptom is rising memory across steps, which is a leak, not fragmentation, and no allocator setting will save you. Step one of the diagnostic sequence exists precisely to separate these.

Skipping the warmup and then blaming variance. If the first batch a job ever sees is a short one, the allocator sizes its initial segments for short batches. The long batch that arrives at step 700 has to find contiguous space in an already-carved heap. Deliberate max-shape warmup is nearly free and removes an entire class of intermittent, hard-to-reproduce OOMs — the kind that only appear when a particular long example lands late in an epoch.

Decision framework: when to choose what

Pick the intervention by the shape of your workload, not by what sounds most sophisticated.

Static shapes, single process, training. You should barely fragment at all. If you do, the cause is almost certainly the eval loop or checkpointing rather than the training step. Fix the outlier path and stop. Do not tune the allocator.

Variable-length sequences, training. Bucketing and length-sorted batching is the primary fix and it is worth the pipeline work. Pair it with max-shape warmup. Reach for expandable_segments:True as a complement, not a substitute — it makes the remaining variance cheaper to absorb but it does not remove the variance.

Production inference, fixed model, predictable batch shapes. This is the case where graph capture and pre-allocated arenas genuinely shine. Capture the execution graph once with fixed shapes and fixed buffers, and per-request allocation churn essentially disappears; you also pick up a latency win from eliminating launch overhead. The constraint is rigidity — any change to shape or model requires re-capture, so it fits stable serving paths and fights you during rapid iteration.

Production inference, dynamic batching, variable request shapes. Graph capture per shape gets combinatorially awkward. Better to define a small set of allowed batch shapes at the serving layer, pad requests up to the nearest one, and let the allocator settle into that fixed set. You are pushing the same bucketing idea one layer up into the request scheduler.

Multi-tenant GPU, several models or jobs on one card. Hardware partitioning on data-center parts, or per-process memory fraction caps as the software fallback. The point is isolation: fragmentation you cannot see is fragmentation you cannot debug, and one greedy tenant will otherwise starve every other one. Note that partitioning costs some overhead and leaves a slice of memory unusable — that is the price of predictability.

Multi-GPU distributed training. Collective communication libraries allocate their own buffers, and those live outside your framework's caching allocator. If OOM correlates with gradient synchronization rather than the forward or backward pass, tune the communication library's buffer configuration before touching the framework allocator. Sharded strategies change the picture again: they lower peak memory but increase allocate-free churn as shards are gathered and released, so shape stability matters more, not less.

You are writing custom kernels or working below the framework. The CUDA stream-ordered memory pool APIs give you direct control — create a pool, set a release threshold so freed memory stays in the pool instead of returning to the driver, and allocate from it. Highest control, highest effort, and the pool holds memory other processes cannot use. Only worth it when you own the whole device.

Nothing has worked and you cannot restructure. Restart the process on a schedule. It is unglamorous and it works: a fresh CUDA context has a clean heap. For long-running inference servers, a rolling restart policy triggered on a fragmentation metric is a legitimate operational answer, the same way it is for any long-lived process with heap pressure. Just make sure you are restarting on a measured signal rather than a guess.

How it connects to cost, scheduling, and capacity planning

Fragmentation rarely stays a systems problem for long. It surfaces as a capacity question, and the teams who handle it well are the ones who wire it into the same dashboards as utilization and cost per unit of work.

The connection runs through effective capacity. If your scheduler believes a node has 80 GB free and a job needs 30 GB, it will place the job. If fragmentation means the largest contiguous run is 22 GB, the job lands and dies, the scheduler retries, and you get a crash loop that looks like a scheduling bug. Cluster schedulers reason about *total* device memory because that is what the device reports; they have no visibility into the internal layout of another process's heap. The practical mitigation is to reserve conservatively — request the peak-plus-margin rather than the average — and to keep one workload per device slice wherever the economics allow.

There is a scheduling-policy angle as well. Bin-packing many small jobs onto a large accelerator maximizes paper utilization and maximizes fragmentation risk simultaneously. Running fewer, larger jobs per device wastes some memory but produces far more predictable behavior. Which is correct depends on whether your bottleneck is capacity or reliability, and that is a business decision, not a technical one.

Quantization interacts here in a way worth flagging. Moving weights from 16-bit to 8-bit or 4-bit halves or quarters the weight footprint, which obviously helps. But it also changes the *size distribution* of allocations, and a workload that had reached a stable allocator steady state can start fragmenting again after a precision change. Re-measure reserved-over-allocated after any quantization or precision change rather than assuming the memory story only improved.

Long-running serving processes deserve their own note. A model server that stays up for weeks, handling requests of varying shapes, accumulates exactly the kind of heap debris that a training job restarting nightly never gets the chance to build. Emit the fragmentation ratio as a metric, alert on it, and treat a rising trend the same way you would treat rising heap usage in any other long-lived service — as an operational signal that something upstream changed, usually the request distribution.

Finally, the discipline generalizes. The reason bucketing works, the reason arenas work, and the reason graph capture works are all the same reason: allocators are efficient when the set of sizes they must satisfy is small and repeats. That is true of database buffer pools, game engine frame allocators, and JVM heaps. If you already have intuition for one of those, you have intuition for this. The GPU case is only harsher because there is no swap to hide behind and the hardware is expensive enough that the waste shows up on an invoice.

Related questions

Does mixed precision make fragmentation better or worse?

Both. Half-precision tensors are smaller, so peak memory drops and you gain headroom. But the mix of FP16 activations and FP32 master weights creates two distinct size families in the same heap, and more distinct sizes means more fragmentation pressure. Net effect is usually positive; re-measure rather than assume.

Will restarting the process actually help?

Yes, completely — a new CUDA context starts with an unfragmented heap. It is a real remedy for long-lived inference servers, implemented as a rolling restart triggered on a fragmentation metric. It is a mitigation, not a cure: whatever allocation pattern caused the fragmentation will rebuild it.

How is this different from a memory leak?

A leak means live memory grows monotonically and never returns — you are holding references you should have dropped. Fragmentation means live memory is stable but unusable in large contiguous chunks. Check whether allocated memory returns to baseline between steps; if it does, it is fragmentation.

Do inference and training fragment differently?

Yes. Training churns huge activation tensors every step, so external fragmentation dominates and shape variance is the driver. Inference allocates smaller, more uniform buffers but runs for weeks, so debris accumulates slowly and restart cadence matters more. Graph capture helps inference far more than training.

Can unified or managed memory sidestep the problem?

Not usefully for performance-critical work. Unified memory lets the driver migrate pages between host and device, which avoids the hard OOM, but the migration cost is severe under GPU-heavy access patterns. It is a correctness escape hatch, not a fragmentation strategy.

FAQ

How do I know for certain that fragmentation is my problem?

Look for the signature: an OOM error requesting an amount smaller than your free memory, plus a large gap between reserved and allocated memory in the framework's own counters. PyTorch's OOM message reports reserved-but-unallocated bytes directly. If that number comfortably exceeds the failed request, the memory exists and the contiguity does not — that is fragmentation.

What is the single highest-value change if I can only make one?

Eliminate shape variance. Bucket your variable-length inputs into a handful of padded sizes and sort batches so each batch is internally uniform. Every other technique — allocator tuning, expandable segments, graph capture — is either working around size variance or exploiting its absence. Removing the variance addresses the cause.

Should I set PYTORCH_CUDA_ALLOC_CONF on every job by default?

No. expandable_segments:True is broadly safe and often helps, so it is a reasonable default to test. max_split_size_mb is not — the right value depends on your allocation size distribution, and a poor choice inflates memory use by preventing legitimate reuse of large cached blocks. Measure before and after on your actual workload.

Does calling empty_cache() help or hurt?

It hurts in a training loop. Releasing cached blocks forces expensive synchronizing driver allocations to reacquire them, costing throughput, and the reacquired blocks are not guaranteed to be better positioned. Its legitimate use is handing the device to another process or model, where you genuinely want the memory returned rather than cached.

Why does my job run fine for hours and then OOM?

Almost always a rare shape or a rare code path. A long sequence arriving late in an epoch, the first evaluation pass, or a checkpoint save can all request a size the heap has never had to satisfy. Warming up at your maximum expected shape before real work starts removes most of this class of failure.

Is fragmentation worse on smaller consumer cards than on large data-center GPUs?

Effectively yes, because the ratio matters more than the absolute number. A 2 GB tensor is a small fraction of an 80 GB heap and easy to place; on a 24 GB card it is a large fraction and needs a substantial contiguous run. Smaller total memory means fewer viable placements for any given large allocation.

Sources

• PyTorch CUDA semantics and memory management
• PyTorch memory snapshot and visualization tooling
• torch.cuda memory API reference
• NVIDIA CUDA C++ Programming Guide — memory pools and stream-ordered allocation
• NVIDIA CUDA Runtime API — memory pool functions
• TensorFlow GPU guide — limiting and growing GPU memory
• NVIDIA Nsight Systems documentation
• NVIDIA Multi-Instance GPU user guide
• JAX GPU memory allocation guide

Related on PULSE

• [How do you measure and improve GPU utilization?](/knowledge/ai433)
• [How do you reduce GPU costs when serving large language models?](/knowledge/ai343)
• [How do you handle GPU scheduling on Kubernetes for AI workloads?](/knowledge/ai361)
• [How do you choose an inference accelerator: GPU, TPU, or custom silicon?](/knowledge/ai415)
• [The 10 Best GPU Cloud Providers for AI Training in 2027](/knowledge/ai340)
• [The 10 Best GPU Orchestration Tools for Kubernetes in 2027](/knowledge/ai358)