What is distributed training and when do you need it?

What distributed training is and why it matters

Distributed training is the practice of running a single training job across many accelerators — several GPUs in one server, or many servers wired together — while keeping the math equivalent to what one enormous device would have computed. The core problem it solves is that a training step has three memory consumers that all scale with model size: the parameters themselves, the gradients, and the optimizer states. For a model trained in mixed precision with the Adam optimizer, the rough accounting is 2 bytes per parameter for the FP16/BF16 weights, 2 bytes for the gradient, and roughly 12 bytes for optimizer state (FP32 master weights plus two moments) — about 16 bytes per parameter before you have allocated a single activation. A 7-billion-parameter model therefore needs on the order of 112 GB just for state, which no single 80 GB accelerator holds. Activations add more on top, scaling with batch size and sequence length.

There are two distinct reasons to distribute, and conflating them is the most common planning mistake. The first is throughput: the model fits, but one device processes too few samples per second, so a run that should take two days takes three weeks. The second is capacity: the model does not fit at all, and no batch size makes it fit. Throughput problems are solved by data parallelism, where every device holds a full replica and processes a different slice of the batch. Capacity problems require sharding — splitting the parameters, gradients, or optimizer states themselves across devices, or splitting individual layers.

Why this matters commercially is straightforward: training time is rented compute, and rented compute is a direct line item. A team that halves wall-clock time on the same hardware count has halved that line item. A team that gets a model into production two months earlier captures two months of whatever revenue or cost avoidance the model was built to produce. But the inverse is equally true — distribution is not free. Every synchronization point adds communication that competes with computation, and a poorly configured 16-GPU job can be slower per dollar than a well-tuned 4-GPU job. The decision to distribute should always be preceded by measurement, not assumption.

The vocabulary is worth pinning down because vendors use the same words differently. Data parallelism replicates the model and splits the batch; gradients are averaged across replicas with an all-reduce collective after every backward pass. Sharded data parallelism — PyTorch's FSDP, or DeepSpeed's ZeRO stages — keeps the data-parallel structure but shards the optimizer states (stage 1), gradients (stage 2), and parameters (stage 3) across ranks, gathering them just in time for each layer's forward and backward. Tensor parallelism splits individual matrix multiplications across devices, so a single attention head or MLP block is computed cooperatively. Pipeline parallelism assigns different layer groups to different devices and streams micro-batches through them. Sequence or context parallelism splits the sequence dimension, which matters for very long context windows where activation memory dominates.

The step-by-step process for standing up a distributed run

Getting from a single-GPU script to a healthy multi-node job follows a predictable order, and skipping steps is how teams end up debugging a 64-GPU job that was never correct on two.

Step one — profile the single-device baseline. Before touching any distributed API, record peak memory with torch.cuda.max_memory_allocated(), measure samples per second, and note GPU utilization. If utilization sits below 80 percent, the bottleneck is the data loader or preprocessing, and adding GPUs will multiply an idle-waiting problem rather than solve it. Fix the input pipeline first: more worker processes, prefetching, pinned memory, and on-GPU augmentation.

Step two — exhaust single-device memory optimizations. Gradient checkpointing (recomputing activations during backward rather than storing them) commonly cuts activation memory by 60–80 percent at a cost of roughly 20–30 percent extra compute. Mixed precision with BF16 halves weight and gradient memory. Gradient accumulation lets you simulate a large effective batch with a small per-step batch. Parameter-efficient fine-tuning — LoRA or QLoRA — trains a small number of adapter parameters instead of the full model and routinely brings a 7B or 13B fine-tune onto one 24–48 GB card. Many jobs that look like they need distribution do not, once these are applied.

Step three — scale to multiple GPUs on one node. Single-node multi-GPU is dramatically simpler than multi-node because the interconnect is NVLink or PCIe inside one chassis, there is no network fabric to misconfigure, and launch is a single process-spawn command. Use plain distributed data parallel if the model fits per device; use FSDP or ZeRO if it does not. Verify that loss curves match the single-GPU baseline for the first few hundred steps — a diverging curve here almost always means a batch-size or learning-rate scaling error, not a framework bug.

Step four — measure scaling efficiency before adding nodes. Scaling efficiency is throughput on N devices divided by N times throughput on one device. Above 0.90 is healthy for single-node data parallelism on well-sized models. If you see 0.6 at eight GPUs, adding a second node will make it worse, not better. Diagnose whether the loss is in the all-reduce (communication-bound) or in the loader (input-bound) before scaling further.

Step five — go multi-node, and treat the network as a first-class dependency. Multi-node introduces a fabric that can silently fall back to a slow path. Confirm the collective library is actually using the high-speed interconnect rather than TCP over the management network — a job that quietly falls back can lose most of its expected speedup while reporting no errors. Set explicit rendezvous configuration, pin the correct network interface, and run a synthetic all-reduce benchmark before the real job.

Step six — checkpoint aggressively and prove recovery works. At scale, hardware failure is a scheduled event, not an exception. Write checkpoints on a fixed interval, and actually restart from one before you commit to a long run. A checkpoint you have never restored is not a checkpoint.

Costs, timelines, and the ranges to plan against

The economics of distributed training are dominated by two variables: the hourly rate of the accelerators and the scaling efficiency you actually achieve. High-end data-center GPU instances on major clouds are typically rented by the node — a full eight-GPU node is the common unit — and hourly rates for current-generation hardware run into the tens of dollars per node-hour on demand. Preemptible or spot capacity generally discounts that substantially, often by half or more, in exchange for the risk of interruption. Reserved or committed-use contracts sit between the two. Because the exact rates change frequently and differ by region and generation, price the specific instance family in your region rather than working from a remembered number, then build the estimate from node-hours.

The estimate itself is simple arithmetic once you have a throughput measurement. Take measured tokens or samples per second per GPU at your target configuration, multiply by GPU count and by your scaling efficiency, divide the total training corpus by that number, and you have wall-clock seconds. Then multiply node-hours by the rate. The dangerous part of this calculation is scaling efficiency: an estimate built on a naive linear assumption at 32 GPUs, when the real figure is 0.7, is off by more than 40 percent in cost and time.

Engineering time is the cost line teams routinely omit. Moving from a working single-GPU script to a stable single-node multi-GPU job is usually a matter of days for an experienced practitioner using a high-level framework. Multi-node is a different tier — first-time multi-node setups reliably consume one to three weeks of an engineer's time on cluster configuration, network validation, container images, storage mounts, and the debugging of hangs that produce no error message. Adding tensor and pipeline parallelism for very large models is a further step up, involving parallelism-degree tuning, micro-batch sizing, and recomputation policy, and it is realistic to budget several weeks of iteration before the configuration is efficient. At an engineer's fully loaded cost, that labor can easily exceed the compute bill for a mid-sized run — which is a strong argument for choosing the simplest parallelism strategy that clears your memory and time constraints.

Typical timelines by tier, assuming the data pipeline is already sound: a supervised fine-tune of a small model on a single GPU runs in hours to a couple of days. A parameter-efficient fine-tune of a mid-sized model on one node runs in hours to days. A full-parameter fine-tune of a large model on one or two nodes runs in days to a couple of weeks. Pretraining from scratch is a fundamentally different undertaking — a multi-week to multi-month commitment of a large cluster, and one that most organizations should not attempt when a fine-tune of an existing checkpoint reaches the same business outcome for a fraction of the cost.

Storage and data movement deserve a line in the budget too. Checkpoints for large models are large — full optimizer state can be several times the size of the weights — and writing them every fifteen to thirty minutes across a long run produces meaningful storage volume and I/O load. If checkpoint writes block training, they show up as a throughput regression that looks like a communication problem. Use asynchronous or sharded checkpointing where the framework supports it, and keep a retention policy so you are not paying to store hundreds of intermediate states.

Finally, plan a benchmarking budget. Spending a few hundred dollars of compute on short calibration runs — sweeping parallelism degrees, micro-batch sizes, and checkpointing settings for a few hundred steps each — routinely saves multiples of that on the full run. Teams that skip calibration and launch straight into a multi-week job are the ones who discover at day nine that a different configuration would have finished in five days.

Where teams get distributed training wrong

Distributing before optimizing. The single most expensive error is adding hardware to a job that was never efficient on one device. If the data loader starves a single GPU, eight GPUs will be starved eight ways. Always fix utilization first; the fix is usually cheap and sometimes eliminates the need to distribute entirely.

Ignoring the effective batch size when scaling. Data parallelism multiplies the effective batch by the number of replicas. Going from one GPU to eight with the same per-device batch means an 8× larger global batch, which changes optimization dynamics. Without adjusting the learning rate and warmup schedule, loss curves diverge or plateau, and teams misattribute the failure to the distributed framework. Scale the learning rate deliberately and use a longer warmup at large global batch sizes.

Assuming linear scaling. Communication cost grows with the number of participants and with parameter count. All-reduce over gradients is a fixed cost per step that does not shrink as you add devices, so at some point adding GPUs adds communication faster than it removes computation. Every configuration has a point of diminishing returns; find yours empirically rather than assuming it is beyond your scale.

Treating the interconnect as an afterthought. Multi-node scaling is a network problem wearing a machine-learning costume. Job placement matters — replicas spread across racks or availability zones can perform dramatically worse than co-located ones. The failure mode is silent: everything runs, nothing errors, and throughput is a fraction of expectations.

Never testing checkpoint restore. Long runs fail. Nodes fail, spot instances are reclaimed, drivers crash. A team that checkpoints but has never restored from one discovers at hour 60 that the checkpoint omits the optimizer state or the data-loader position, and the run must restart from zero.

Choosing maximum parallelism for a model that does not warrant it. Tensor and pipeline parallelism exist because some models genuinely cannot be trained any other way. Applying them to a model that fits comfortably under sharded data parallelism adds configuration surface, debugging difficulty, and often worse throughput because tensor parallelism demands very high-bandwidth links and degrades badly across node boundaries. Keep tensor parallelism within a node wherever possible.

Non-determinism ambushes. Distributed runs introduce ordering effects in reductions and data sharding that make exact reproduction harder. If reproducibility matters for compliance or debugging, set seeds per rank deliberately, record the full parallelism configuration alongside each checkpoint, and accept that bitwise reproducibility across different device counts is generally not achievable.

Forgetting that evaluation also distributes. Teams carefully shard training and then run evaluation on rank zero only, creating a synchronization stall where seven of eight GPUs idle. Distribute evaluation, or run it asynchronously from checkpoints on separate hardware.

Skipping observability. Without per-rank throughput, GPU utilization, memory, and time-in-collective metrics, debugging a slow job is guesswork. Instrument before you scale, not after the run is already burning money.

Decision framework: choosing a strategy and a tool

The decision reduces to a short sequence of questions answered with measurements, not opinions.

Does the model plus optimizer state plus activations fit on one device, with your target batch size, after mixed precision and gradient checkpointing? If yes, and throughput is acceptable, do not distribute. This is the correct answer far more often than the discourse suggests, particularly for fine-tuning workloads where adapter-based methods keep trainable parameters tiny.

If it fits but is too slow — use data parallelism. Standard distributed data parallel is the simplest, most-tested path in every major framework. Scale within one node first. Expect strong efficiency for compute-heavy models; expect weaker efficiency for small models where the gradient all-reduce dominates each step.

If it does not fit but is within roughly an order of magnitude of a single device's capacity — use sharded data parallelism. FSDP in PyTorch and ZeRO stage 2 or 3 in DeepSpeed both shard optimizer state, gradients, and optionally parameters across the data-parallel group. This is the highest-leverage tier: it keeps the mental model of data parallelism while multiplying effective capacity roughly with device count. Enabling CPU or NVMe offload extends capacity further at a significant throughput cost — useful when the alternative is not training at all, wasteful when it is not.

If it still does not fit, or the layer itself is too large — add tensor parallelism inside a node and pipeline parallelism across nodes. This is the regime where purpose-built frameworks earn their complexity. Tensor parallelism is bandwidth-hungry and should stay within a single node's high-speed domain; pipeline parallelism tolerates slower links but requires careful micro-batch sizing to keep the pipeline bubble small.

Tool selection follows the strategy, not the reverse. For PyTorch teams that want minimal new concepts, native distributed data parallel and FSDP are the default and require no additional dependency. Hugging Face Accelerate wraps the same primitives with far less boilerplate and is the fastest path from a single-GPU script to a multi-GPU one. DeepSpeed offers the widest tuning surface for ZeRO stages and offload, at the cost of a substantial configuration file. Frameworks in the Megatron and NeMo lineage provide production-grade tensor, pipeline, and sequence parallelism for very large models. PyTorch Lightning and Ray Train sit at a higher level, trading some low-level control for orchestration, checkpointing, and fault tolerance you would otherwise build yourself. For TPU targets, the JAX ecosystem is the practical choice. Horovod remains relevant mainly for mixed-framework or MPI-centric environments.

The organizing principle: pick the least complex tier that clears your constraint, and only move up a tier when you have measured evidence that the current tier cannot get there.

Related questions

Does distributed training change my model's accuracy?

It should not, if configured correctly — the math is equivalent. What does change accuracy is the larger effective batch size that data parallelism produces, which requires learning-rate and warmup adjustment. Divergence after scaling is almost always an optimizer-schedule issue, not a correctness bug in the framework.

How many GPUs do I actually need to start?

Two on the same machine is enough to validate correctness and measure scaling. Never debug a distributed setup at large scale first. Confirm the loss curve matches your single-GPU baseline on two devices, then scale within the node, then across nodes.

Can I run distributed training on preemptible or spot capacity?

Yes, and it is a major cost lever, but only with working fault tolerance. Checkpoint on a fixed interval, verify restore end to end, and use an orchestrator that relaunches on preemption. Without that, a reclaimed node discards every hour since the last successful checkpoint.

Is distributed training necessary for fine-tuning?

Frequently not. Parameter-efficient methods such as LoRA and QLoRA train a small fraction of parameters and fit substantially larger models onto a single device. Full-parameter fine-tuning of large models is where distribution becomes genuinely necessary.

What does poor scaling efficiency usually indicate?

Most often the data pipeline is starving the GPUs, or the interconnect has silently fallen back to a slow transport. Check GPU utilization per rank and time spent in collective operations before assuming the framework or the model is at fault.

FAQ

When do I know it is time to move from one GPU to many?

Two triggers. First, capacity: you hit out-of-memory errors even at batch size one after enabling mixed precision, gradient checkpointing, and adapter-based tuning. Second, time: your measured throughput implies a wall-clock duration longer than the project can tolerate. If neither applies, single-device training is simpler, cheaper to debug, and fully reproducible.

What is the difference between FSDP and DeepSpeed ZeRO?

Both shard optimizer states, gradients, and parameters across the data-parallel group so aggregate memory scales with device count. FSDP is native to PyTorch and integrates with the standard tooling with less configuration. DeepSpeed exposes more knobs, including staged sharding levels and CPU/NVMe offload, which matters most at the very largest model sizes. For mid-sized models the two land in broadly similar territory; pick based on which ecosystem your team already runs.

Why did my job hang with no error message?

Hangs in distributed training are usually collective mismatches: one rank enters an all-reduce that another rank never reaches, often because of a conditional branch that differs per rank, an uneven final batch, or evaluation running on rank zero only. Enable collective timeout diagnostics, log which rank reaches which barrier, and confirm every rank executes the same sequence of collectives.

Do I need a specialized high-speed network, or is standard Ethernet enough?

It depends on strategy. Sharded and tensor-parallel strategies move large volumes of parameter data every step and are highly sensitive to interconnect bandwidth. Plain data parallelism moves gradients once per step and tolerates slower links better, especially with gradient accumulation reducing sync frequency. Within a single node, the internal GPU-to-GPU links are the relevant fabric; across nodes, a high-bandwidth RDMA-capable network is what separates good scaling from poor scaling.

How often should I checkpoint?

Frequently enough that the work lost to a failure is tolerable, and infrequently enough that writes do not dominate. A common practice is every fifteen to thirty minutes of training time, using asynchronous or sharded checkpointing so the write overlaps computation. On preemptible capacity, shorten the interval. Always keep at least one verified-restorable checkpoint and test the restore path before the run is long.

Can distributed training reduce cost, or only time?

It can do both, but not automatically. If scaling efficiency stays high, you use roughly the same total GPU-hours in less wall-clock time, so cost is flat and time drops — which is often worth real revenue through earlier delivery. If efficiency is poor, you burn more GPU-hours for the same result and cost rises. The lever that genuinely reduces cost is a strategy that lets you use cheaper or fewer accelerators, such as sharding to avoid a larger instance class, or preemptible capacity with reliable checkpointing.

Sources

• PyTorch Distributed Data Parallel tutorial
• PyTorch FSDP documentation
• Hugging Face Accelerate documentation
• DeepSpeed project site
• DeepSpeed GitHub repository
• NVIDIA NeMo Framework user guide
• NVIDIA Megatron-LM repository
• JAX documentation
• Ray Train documentation
• PyTorch Lightning documentation

Related on PULSE

• [How do you build data pipelines for continuous model training?](/knowledge/ai403)
• [The 10 Best Distributed Training Frameworks in 2027](/knowledge/ai386)
• [The 10 Best GPU Cloud Providers for AI Training in 2027](/knowledge/ai340)