What is continuous batching and how much more throughput does it give an LLM server in 2027?

A concrete scenario: the static-batch trap

Picture a production inference server behind a chat product. Traffic arrives unevenly: some prompts are 40 tokens, some are 4,000; some responses end in 20 tokens, others run to 1,500. A naive server groups requests into a fixed batch, runs the whole batch to completion, then returns every response at once and starts the next batch. That design has a structural flaw: the batch is only as fast as its slowest member. If one request in a batch of 32 generates 1,500 tokens while the other 31 finish in 60, the GPU keeps computing attention for 31 finished sequences for another 1,440 decode steps. Those slots are wasted — the hardware is busy but producing nothing useful for anyone.

The waste compounds. Decode is memory-bandwidth-bound, not compute-bound. Each step reads the model weights plus the KV cache for every sequence in the batch. If 31 of 32 sequences are done, you are still paying the full weight-read cost per step while emitting one useful token. Utilization looks high on a dashboard, but goodput — tokens actually delivered to users — collapses. In practice, static batching on mixed-length chat traffic often lands at 15-30% of the throughput the same GPU could sustain, and tail latency is terrible because every user waits for the longest generation in their batch.

The fix is not a faster kernel. It is a scheduling change: stop treating the batch as an immutable unit and start treating it as a slot pool that gets refilled continuously. That is exactly what continuous batching does, and it is the single highest-leverage change most teams make to an LLM server before they touch quantization or speculative decoding.

How the mechanism actually works

Continuous batching — sometimes called iteration-level scheduling — decouples the batch from the request lifecycle. The server maintains a fixed number of execution slots (bounded by KV-cache memory, not by a request count). At each decode iteration, the scheduler decides which sequences occupy those slots for the next step.

The loop, concretely:

1. Prefill admission. New requests arrive in a queue. The scheduler picks some to prefill. With chunked prefill, a long prompt is split into chunks of, say, 512 or 1,024 tokens so a 4,000-token prompt does not monopolize a full step and stall every decoding sequence.
2. Mixed step. The engine builds one batch containing both prefill chunks and decode tokens. Prefill chunks are compute-bound; decode tokens are memory-bound. Mixing them raises arithmetic intensity and keeps both tensor cores and memory controllers busy.
3. Token emission. Every decoding sequence in the batch produces exactly one token this step. Sequences that hit an end-of-sequence token or their length cap are marked finished.
4. Eviction and refill. Finished sequences release their slots and their KV-cache blocks immediately. The scheduler pulls the next waiting requests into the freed slots for the very next step.
5. Repeat. No drain phase, no barrier, no waiting for the slowest member. The batch composition changes every step.

The critical enabler is paged KV cache (the PagedAttention idea). KV cache is allocated in fixed-size blocks rather than one contiguous buffer per sequence. Blocks are referenced through a page table, so a sequence can grow without pre-reserving its maximum length, and freed blocks return to a pool instantly. Without paging, memory fragmentation would force the server to reserve worst-case KV for every admitted request, capping concurrency far below what the GPU can actually hold.

Two knobs dominate real-world behavior. Max batch size (or max concurrent sequences) sets the ceiling on how many slots exist; too low and you leave GPU idle, too high and you thrash the KV cache and blow up latency. Chunked prefill size sets how much prompt work enters each step; large chunks improve prefill throughput but add latency spikes to decoding sequences sharing the step. Production servers tune these per model and per GPU — a 7B model on one 80GB accelerator supports far more concurrent sequences than a 70B model on the same card.

Real numbers, ranges, and benchmarks

The honest answer to "how much more throughput" is a range, because the gain depends on how bad your baseline is and how skewed your traffic is. Published results from the vLLM team and subsequent academic work (Orca, vLLM, Sarathi-Serve, and the continuous-batching literature) consistently show the following shape:

• Versus naive static batching: 5x to 20x throughput improvement is the commonly reported band. The high end appears with highly variable output lengths, which is exactly the chat and agent traffic most teams run. The low end appears with uniform, short, fixed-length workloads where static batching was never that wasteful.
• Versus a well-tuned dynamic batcher with a short timeout: 2x to 4x. If your baseline already waits only a few milliseconds to form batches and caps batch size sensibly, continuous batching still wins because it eliminates the drain phase, but the multiple is smaller.
• Latency under load: time-to-first-token often improves because prefill is no longer blocked behind a long decode batch, while inter-token latency can degrade slightly as concurrency rises, since more sequences compete for the same memory bandwidth per step.
• Goodput at fixed SLO: this is the metric that matters. Teams commonly report serving 3x to 8x more requests per second at a fixed p95 latency target after switching, because the scheduler can hold more concurrent sequences without the tail blowing out.

For 2027 hardware specifically, several trends push the multiple upward: larger and faster HBM makes KV cache cheaper per token, so more sequences fit; FP8 and FP4 inference widen the compute-to-bandwidth ratio, which makes the memory-bound decode phase relatively more expensive and therefore makes slot efficiency more valuable; and disaggregated prefill/decode architectures let a dedicated prefill pool feed a decode pool that runs continuous batching at very high concurrency. The mechanism itself has not changed, but the ceiling it unlocks keeps rising.

A rough planning heuristic: if your current server is static-batched, assume a 6x to 10x throughput gain from continuous batching alone before any other optimization. If you already run a modern engine with continuous batching enabled, further gains come from tuning max concurrency, chunked prefill size, and KV-cache block size — typically another 20% to 60%, not another order of magnitude. Do not let anyone promise 20x on top of an already-optimized stack; that number describes replacing a bad baseline, not improving a good one.

Trade-offs and alternatives

Continuous batching is not free. It trades simplicity and predictable per-request latency for aggregate throughput. The main costs:

• Fairness and starvation. A scheduler that always admits the shortest waiting request can starve long prompts. Most engines use FIFO admission with chunked prefill to bound this, but you should still monitor queue time by request class.
• Latency variance. Because batch composition changes every step, inter-token latency jitters. Real-time voice or streaming UX may need a concurrency cap or a priority lane rather than maximum throughput.
• Memory pressure. Higher concurrency means more KV blocks in flight. If you overshoot, the engine preempts sequences (swapping or recomputation), which can cost more than the concurrency gained.
• Complexity. Iteration-level scheduling, paged memory, and chunked prefill are harder to debug than a fixed batch loop. Observability — per-step batch size, KV utilization, preemption count — becomes mandatory.

Alternatives and complements worth knowing:

• Static batching remains fine for offline batch jobs with uniform lengths, like bulk classification or embedding generation. Its simplicity is a feature there.
• Dynamic batching with a timeout is the middle ground: wait N milliseconds to accumulate requests, then run them as a batch. It helps at low load but still drains.
• Speculative decoding reduces the number of forward passes needed per token and stacks multiplicatively with continuous batching.
• Disaggregated prefill/decode separates the two phases onto different pools so each can be tuned independently; continuous batching runs on the decode pool.
• Quantization (FP8/INT8/FP4) shrinks weights and KV, letting more sequences fit — which directly raises the concurrency ceiling continuous batching can exploit.

The practical rule: use continuous batching whenever request lengths vary and traffic is online. Use static batching when lengths are uniform and latency does not matter. Layer the complements only after the scheduler is tuned, because a badly tuned continuous batcher can underperform a simple dynamic one.

Common pitfalls and how to avoid them

Pitfall 1: cranking max concurrency to the moon. More concurrent sequences is not free throughput; past a point, KV cache pressure forces preemption and every preemption costs a recompute or a swap. Watch KV-cache utilization and preemption rate. If preemptions climb, lower concurrency or shrink block size.

Pitfall 2: ignoring chunked prefill. Without it, a single long prompt can occupy an entire step and stall all decoding sequences, producing latency spikes that look like a scheduler bug. Enable chunked prefill and tune the chunk size — 512 to 2,048 tokens is a common starting range.

Pitfall 3: benchmarking with uniform synthetic prompts. If every request is 128 tokens in and 128 out, continuous batching looks barely better than static batching, and you will draw the wrong conclusion. Benchmark with a realistic length distribution, ideally replayed from production traces.

Pitfall 4: measuring throughput instead of goodput. Tokens per second at 100% GPU utilization means nothing if p99 latency is 30 seconds. Track requests per second at a fixed latency SLO, plus time-to-first-token and inter-token latency percentiles.

Pitfall 5: assuming the gain is automatic. Continuous batching must actually be enabled and configured. Some serving stacks default to it; others require explicit flags. Confirm from the engine's metrics that batch composition is changing step to step — if batch size is constant across a request's lifetime, you are not running continuous batching.

Pitfall 6: forgetting the client side. A server that admits requests continuously still needs clients that stream tokens. If your client waits for the full response, you have moved the bottleneck, not removed it.

Pitfall 7: no capacity model. Before and after any change, model your tokens-per-second per GPU, your peak concurrency, and your KV memory per sequence. Without that, you cannot tell whether a regression is the scheduler, the model, or the traffic.

Related questions

Does continuous batching change model output quality?

No. It changes scheduling, not math. Each sequence still attends only to its own KV cache, so outputs are identical to static batching at the same sampling settings, assuming the same numerical precision.

What is the difference between continuous batching and dynamic batching?

Dynamic batching waits a short timeout to group requests, then runs the group to completion. Continuous batching refills slots at every decode step, so no sequence waits for the slowest member of its batch.

How much KV cache memory does each concurrent sequence need?

Roughly 2 × layers × KV heads × head dim × sequence length × bytes per element. For a 70B model in FP8 with grouped-query attention, a 4,000-token sequence can need on the order of a few hundred megabytes.

Can continuous batching run on a single GPU?

Yes. It is a scheduling and memory-management technique, not a distributed one. A single accelerator with paged KV cache and a continuous scheduler captures most of the benefit.

Does it help prefill-heavy workloads?

Less directly. Prefill is compute-bound and already efficient in large batches. Chunked prefill helps latency, but the biggest throughput wins from continuous batching come from decode-heavy, mixed-length traffic.

FAQ

How much more throughput does continuous batching give in 2027? Against naive static batching, expect 5x to 20x on mixed-length online traffic, with 6x to 10x a reasonable planning number. Against a well-tuned dynamic batcher, expect 2x to 4x. The exact figure depends on your length distribution, concurrency limits, and KV-cache headroom.

Is continuous batching the same as PagedAttention?

No, but they are complementary. Continuous batching is the scheduling policy that refills slots every step. PagedAttention is the memory manager that makes it practical by allocating KV cache in blocks so slots can be freed and reused instantly.

What limits how many sequences can run at once? KV-cache memory is almost always the binding constraint, not compute. Once KV blocks are exhausted, the engine either queues, preempts, or swaps. That is why quantization and grouped-query attention raise the effective concurrency ceiling.

Does continuous batching hurt latency? It can raise inter-token latency slightly at high concurrency because more sequences share memory bandwidth per step. Time-to-first-token often improves. Cap concurrency or use priority lanes if you have strict real-time SLOs.

Do I need to change my model to use it? No. It works with standard transformer decoders. You need a serving engine that implements iteration-level scheduling and paged KV cache, plus enough GPU memory to hold the concurrency you want.

How do I verify it is actually working? Check engine metrics for per-step batch size variation, KV-cache utilization, preemption count, and queue time. If batch size is fixed for a request's entire lifetime, continuous batching is not active.

Sources

• vLLM documentation and PagedAttention paper: https://docs.vllm.ai/
• Orca paper on iteration-level scheduling: https://www.usenix.org/conference/osdi22/presentation/yu
• Sarathi-Serve (chunked prefill) paper: https://arxiv.org/abs/2403.02310
• NVIDIA TensorRT-LLM documentation: https://docs.nvidia.com/tensorrt-llm/
• Hugging Face Text Generation Inference docs: https://huggingface.co/docs/text-generation-inference
• PyTorch blog on LLM inference: https://pytorch.org/blog/
• Efficient Memory Management for Large Language Model Serving (vLLM paper): https://arxiv.org/abs/2309.06180

Related on PULSE

• How paged KV cache raises LLM server concurrency
• Chunked prefill explained for latency-sensitive chat
• Speculative decoding: when it stacks with continuous batching
• Disaggregated prefill and decode architectures
• Quantization choices that change your concurrency ceiling
• Measuring goodput instead of GPU utilization for LLM serving