How do you scale LLM inference to handle thousands of concurrent users?

The outcome you should expect

The single most important shift when you move from a demo to a production inference tier is that throughput stops being a property of the GPU and starts being a property of the *scheduler*. A naive implementation — one request in, one request out, padded to the longest sequence in a static batch — will leave a modern datacenter GPU somewhere in the 10–25% utilization range. Every token you generate for user A leaves the tensor cores idle while user B waits in a queue. Fixing that scheduling problem, not buying more hardware, is where the first order-of-magnitude improvement lives.

With continuous batching (also called in-flight batching or rolling batching), the server admits new requests into an already-running batch at token boundaries instead of waiting for the whole batch to finish. A user who submits a 30-token prompt does not sit behind a user generating a 2,000-token essay. Published results from the vLLM project and NVIDIA's TensorRT-LLM both report multiple-times throughput gains versus static batching on the same hardware, with the exact multiple depending heavily on how variable your output lengths are. If every request in your workload generates exactly 100 tokens, continuous batching buys you relatively little. If your output length distribution has a long tail — which nearly every chat product does — it buys you a great deal.

The second shift is memory. A 70B-parameter model in FP16 needs roughly 140 GB just for weights, before a single user connects. KV cache is what actually limits concurrency: each active sequence holds per-layer key and value tensors proportional to its context length. On a 70B model with a large context, per-sequence KV cache can run into the hundreds of megabytes to low gigabytes. Multiply by the number of simultaneous sessions and you see why memory, not compute, caps how many concurrent users a replica can hold. PagedAttention-style allocators — vLLM's core contribution, since adopted in various forms across the ecosystem — break KV cache into fixed-size blocks so you stop reserving worst-case contiguous memory for every sequence. The practical effect is that you fit substantially more simultaneous sequences in the same VRAM.

So the realistic outcome to expect: a well-configured cluster of eight 80GB-class GPUs, serving a 70B model with tensor parallelism and continuous batching, sustaining low-thousands of concurrent chat sessions. Aggregate output throughput lands in the low thousands of tokens per second, time-to-first-token in the low hundreds of milliseconds, and inter-token latency comfortable enough that a streaming UI reads as instantaneous. Push past that and you scale horizontally: add replicas behind a router, not more layers of tuning on one replica.

Set expectations with your stakeholders in *concurrency* terms, not requests per second. "Thousands of concurrent users" is ambiguous — a thousand users with a browser tab open is a very different load from a thousand users with an active generation in flight. The number that matters to the scheduler is concurrent *in-flight sequences*, and in a typical chat product that is a small fraction of logged-in users, often single-digit percent. Measure the real ratio in your own product before you size the fleet, because sizing on total registered users will overspend your infrastructure revenue budget by an order of magnitude.

What drives that outcome

Four levers dominate. Understanding which one is binding at any moment is most of the operational skill in running an inference tier.

Batching policy. Larger batches amortize the cost of streaming weights out of HBM across more sequences, which is why throughput climbs steeply with batch size in the decode phase — decode is memory-bandwidth-bound, not compute-bound. But larger batches also lengthen the per-step time, which raises inter-token latency for everyone in the batch. Most engines expose a maximum number of sequences and a maximum number of batched tokens; tuning those two knobs against your latency SLO is the single highest-leverage configuration task you will do. Expect to sweep them, not to guess them.

Prefill versus decode. These are two entirely different workloads sharing one GPU. Prefill (processing the prompt) is compute-bound and scales with prompt length; decode (generating output) is bandwidth-bound and scales with batch size. A long prompt arriving mid-stream can stall decode for every other user in the batch — the classic "one 100K-token request tanks P99 for everyone" failure. Chunked prefill splits a long prompt into pieces interleaved with decode steps, smoothing that spike. Disaggregated serving goes further and runs prefill and decode on separate GPU pools so they stop interfering entirely; that is more operational complexity, and worth it mainly when your prompt-length distribution is wide.

Parallelism strategy. Tensor parallelism splits each layer across GPUs and needs very fast interconnect — NVLink within a node is fine, Ethernet between nodes is not, and crossing a slow link can add multiples to your latency. Pipeline parallelism splits layers across devices and tolerates slower links but introduces bubbles that hurt latency. The default that works: tensor-parallel within a node up to the node's GPU count, then replicate whole model instances across nodes and load-balance between them. Only reach for cross-node tensor parallelism when the model genuinely does not fit in one node.

Quantization and precision. Moving weights from FP16 to FP8 or 4-bit roughly halves or quarters the memory footprint, which frees VRAM for KV cache and therefore directly raises the concurrency ceiling. It also reduces the bandwidth pressure of decode, so throughput rises. The trade is accuracy: 8-bit formats are generally close to lossless for most tasks; 4-bit weight quantization is usually usable but shows measurable degradation on harder reasoning and long-context tasks. Quantizing the KV cache itself is a separate, and often bigger, concurrency win — KV cache is what runs out first.

The diagram makes the feedback loop explicit: freed KV blocks are what admit the next waiting user. When your queue depth climbs, the fix is almost never "more CPU" — it is either more KV capacity (bigger GPUs, quantized cache, shorter max context) or more replicas.

Benchmarks and realistic ranges

Treat every published number, including the ones below, as a starting hypothesis you must re-measure on your own traffic. Throughput figures vary by 30–50% between a vendor benchmark and a production workload, almost entirely because of prompt and output length distribution.

What to measure. The four numbers that matter are time-to-first-token (TTFT), inter-token latency (ITL, sometimes time-per-output-token), end-to-end request latency, and aggregate output tokens per second. Report all of them at P50, P95, and P99 — a mean latency on an inference tier is close to meaningless because the distribution is heavily skewed by long generations. Track them against a fixed concurrency level, and produce a curve, not a point: throughput versus concurrency, and latency versus concurrency, swept from 1 to well past your target.

Rough anchors for a 70B-class model in FP16 on 80GB-class datacenter GPUs, tensor-parallel across 8 GPUs: aggregate output throughput on the order of low thousands of tokens per second under heavy batching; TTFT in the low hundreds of milliseconds for typical prompts of a few hundred to a few thousand tokens; ITL in the tens of milliseconds, which streams comfortably faster than a person reads. Concurrency in the low thousands of in-flight sequences before latency degrades sharply. Smaller models change these numbers dramatically — an 8B model on a single 80GB GPU serves hundreds of concurrent sequences on its own and is often the correct answer when quality permits.

The knee. Every configuration has a knee in the latency-versus-concurrency curve. Below it, added concurrency is nearly free — throughput rises and latency barely moves. Above it, the batch is saturated, queueing begins, and latency rises roughly linearly with load while throughput plateaus. Find your knee empirically and set your autoscaler to add capacity at roughly 70–80% of it, so you have headroom to absorb a burst during the 3–5 minutes it takes to provision a new GPU node.

How to run the benchmark honestly. Replay real production traffic, or at minimum sample real prompt and output lengths from your logs and generate synthetic load matching that distribution. Use a load generator that models open-loop arrival (Poisson-ish request arrival independent of response time) rather than closed-loop (fixed number of workers, each waiting for its response). Closed-loop benchmarks systematically hide queueing collapse, because slow responses automatically reduce offered load — exactly the opposite of what real users do. Warm the cache and discard the first minute. Run for at least ten minutes at steady state before recording percentiles.

Cost math. Divide your fully-loaded hourly GPU cost by measured tokens per second per hour to get cost per million tokens; that number, not the sticker price of the instance, is what belongs in your unit-economics model. Utilization dominates it — a cluster running at 30% average utilization costs three times per token what the same cluster costs at 90%. This is why the batching and autoscaling work pays for itself so quickly: it converts idle GPU time directly into gross margin.

Risks, edge cases, and failure modes

KV cache exhaustion and preemption thrash. When memory runs out mid-generation, engines either preempt a sequence (dropping its cache and recomputing later) or swap it to host memory. Under sustained overload this becomes thrash: the same sequences get preempted repeatedly, work is recomputed, effective throughput collapses while GPU utilization still reads high. Watch the preemption counter your engine exposes; a nonzero, rising preemption rate is an early warning that fires well before user-visible latency does. Cap max context length per request and cap total in-flight sequences so the scheduler refuses work rather than accepting it and thrashing.

Long-context requests as noisy neighbors. A single request with a very long prompt occupies KV blocks proportional to its length and monopolizes prefill compute. In a shared batch it degrades everyone. Mitigations: enable chunked prefill, route long-context requests to a separate pool with its own SLO, and enforce a hard token limit at the API gateway rather than discovering it at the model.

Head-of-line blocking in the router. If your load balancer uses round-robin or plain least-connections, it will happily route a new request to a replica whose batch is already saturated while another replica idles. Inference requests have wildly variable service times, so connection count is a poor proxy for load. Use the engine's own queue-depth or KV-utilization metric to drive routing decisions where your proxy supports it, or at minimum use least-outstanding-requests rather than round-robin.

Cold starts. Loading a 70B model's weights into GPU memory takes minutes, and provisioning a new GPU node on a cloud provider takes minutes more. That means your autoscaler cannot respond to a traffic spike in real time — it can only respond to a trend. Keep a warm buffer sized to your largest plausible minute-over-minute jump, pre-pull container images onto nodes, and use fast weight formats to shorten load time. Serverless GPU platforms mitigate but do not eliminate this.

Streaming and connection handling. Thousands of concurrent users means thousands of long-lived streaming HTTP or WebSocket connections. Default proxy configurations frequently buffer responses (destroying the streaming experience) or time out long generations at 30 or 60 seconds. Check buffering, idle timeout, and max connection settings at every hop: CDN, load balancer, ingress, and application server. Also handle client disconnect properly — if a user closes the tab and your server keeps generating, you are burning GPU time on tokens nobody will read. Propagate cancellation all the way to the inference engine.

Retry storms. When latency rises, clients retry; retries add load; load raises latency further. This is the classic congestive collapse pattern and it turns a small degradation into a full outage in under a minute. Defend with a bounded admission queue that fast-fails with a clear 429 when full, exponential backoff with jitter on the client, and a circuit breaker that sheds load rather than queueing it indefinitely. Shedding 5% of requests cleanly is vastly better than making 100% of them time out.

Silent quality regressions. Quantization, speculative decoding, and prefix caching all change what the model outputs or how it is scheduled. Speculative decoding is designed to be output-equivalent when implemented correctly, but a bug in the verification step degrades quality invisibly — throughput looks great and nobody notices the answers got worse. Keep a golden evaluation set and run it against every configuration change, not just model changes.

Nondeterminism confusing debugging. Batch composition affects floating-point reduction order, so identical inputs can produce different outputs depending on what else was in the batch. This is expected behavior, not a bug, but it will consume days of engineering time if the team does not know about it in advance. Document it.

A practical rollout plan

Stage the work so each phase produces a measurable result and a decision point, rather than trying to build the full production topology up front.

Phase 1 — establish the baseline. Deploy one replica of your target model on one node with a production-grade engine. Do not tune anything yet. Run an open-loop load sweep from 1 to several hundred concurrent sequences using replayed traffic. Record the throughput and latency curves. This gives you the knee, the per-replica concurrency ceiling, and the cost per million tokens you are starting from. Budget a few days; most of it is building the load harness, which you will reuse forever.

Phase 2 — tune the single replica. Sweep max batched tokens and max concurrent sequences against your latency SLO. Enable chunked prefill if your prompt lengths vary. Turn on prefix caching if you have a shared system prompt — in a chat product with a long instruction preamble, this often removes a large fraction of prefill work outright. Evaluate quantization: try 8-bit weights first, measure both throughput and your golden eval set, and only consider 4-bit if the concurrency ceiling still binds. Re-run the Phase 1 sweep after each change so you can attribute gains. Expect this phase to produce the largest single improvement of the whole project.

Phase 3 — horizontal scale and routing. Add replicas behind a router. Verify that scaling is close to linear — if two replicas do not deliver nearly twice the throughput, your router or your shared state is the bottleneck, not the GPUs. Configure routing on queue depth or KV utilization rather than connection count. Add the admission queue with fast-fail behavior and verify it by deliberately overloading the tier and confirming you get clean 429s instead of timeouts.

Phase 4 — autoscaling and warm capacity. Wire the autoscaler to queue depth, with scale-up at roughly 70–80% of the measured knee and a deliberately slower scale-down to avoid flapping. Account for cold start time in the trigger threshold: if a node takes four minutes to become ready, your trigger must fire four minutes before you need it. Maintain a warm buffer of at least one replica above steady-state demand.

Phase 5 — observability and game days. Instrument TTFT, ITL, queue depth, batch size, KV utilization, preemption rate, and GPU utilization, with alerts on P99 latency and queue depth rather than on averages. Then deliberately break things in a staging environment: kill a replica mid-generation, submit a maximum-length prompt flood, simulate a retry storm. Each failure mode you rehearse is one you will not diagnose live at 2am.

Throughout, resist the temptation to skip Phase 1. Teams that jump straight to a multi-node cluster invariably cannot tell whether a change helped, because they have no baseline curve to compare against — and they end up paying for hardware that a two-line configuration change would have made unnecessary.

Related questions

How many GPUs do I need for a given concurrency target?

Work backward from memory. Model weights plus per-sequence KV cache times your target in-flight sequences must fit in aggregate VRAM, with headroom. Then confirm the compute side by measuring throughput at that concurrency. Memory almost always binds first on large models.

Does a smaller model solve this more cheaply than better serving?

Frequently, yes. An 8B model serves many times the concurrency of a 70B on identical hardware. Evaluate quality on your actual task first — if a smaller or distilled model meets your bar, that is the cheapest scaling lever available, by a wide margin.

Should I self-host or use a managed inference API?

Managed APIs win below moderate, spiky volume: no cold starts to manage, no GPU reservations, no on-call. Self-hosting wins at sustained high volume where you can keep utilization above roughly 60–70%, or when data residency and model customization are hard requirements.

What is speculative decoding and is it worth enabling?

A small draft model proposes several tokens; the large model verifies them in one pass. It reduces latency when batch sizes are small, but the gain shrinks or reverses at high batch sizes where the GPU is already saturated. Benchmark it at your actual operating concurrency.

How do I handle traffic spikes I cannot provision for?

Shed load deliberately. A bounded admission queue that returns 429 with a retry-after header preserves the experience for admitted users, while unbounded queueing degrades everyone simultaneously. Pair it with a warm buffer and client-side backoff with jitter.

FAQ

Why does my GPU show high utilization but low throughput?

GPU utilization as reported by standard tooling measures whether any kernel is running, not whether the hardware is doing useful work. A memory-bandwidth-bound decode step with a batch size of one will show near-100% utilization while producing a tiny fraction of achievable throughput. Track tokens per second and batch size distribution instead; utilization alone is a misleading metric on inference workloads.

What limits concurrency more — compute or memory?

Memory, in almost every large-model deployment. Weights consume a fixed share of VRAM, and whatever remains is divided among active sequences as KV cache. When that pool is exhausted, the scheduler stops admitting requests regardless of how much compute is idle. This is why KV cache quantization and paged allocation deliver larger concurrency gains than faster GPUs.

Can I run production inference on consumer GPUs?

For smaller quantized models, yes — it is a legitimate cost play. The constraints are limited VRAM per card (which caps both model size and concurrency), the absence of high-bandwidth GPU-to-GPU interconnect (which makes tensor parallelism expensive), and datacenter licensing terms. It suits small models and moderate concurrency, not a 70B model serving thousands of concurrent sessions.

How do I keep costs predictable as usage grows?

Instrument cost per million tokens as a first-class metric alongside latency, and enforce per-tenant rate limits so one customer cannot consume the fleet. Reserved or committed capacity for your steady-state baseline plus on-demand for peaks is usually cheaper than pure on-demand. Track the ratio of inference spend to product revenue monthly — that ratio, not absolute spend, tells you whether scaling is healthy.

Does prefix caching actually help a chat product?

Substantially, when there is a shared prefix. A long system prompt reused across every request, or a multi-turn conversation where the history is resent each turn, means the same tokens get prefilled repeatedly. Caching those KV blocks eliminates that redundant compute. The benefit scales directly with how much of your average prompt is repeated content.

How should I load-test before launch?

Replay real traffic shapes with an open-loop generator, sweep concurrency past your target until latency breaks, and record percentiles at steady state. Test the failure path too: overload deliberately and confirm you get clean rejections rather than timeouts, cascading retries, or crashed replicas. A load test that only proves the happy path has not tested the thing that will page you.

Sources

• vLLM documentation
• NVIDIA TensorRT-LLM
• Hugging Face Text Generation Inference
• NVIDIA Triton Inference Server
• Ray Serve documentation
• SGLang project
• MLCommons MLPerf Inference benchmarks
• Efficient Memory Management for LLM Serving with PagedAttention (arXiv)
• DeepSpeed-MII
• llama.cpp server

Related on PULSE

• [How do you evaluate LLM output quality at scale?](/knowledge/ai367)
• [What causes high latency in LLM inference and how do you fix it?](/knowledge/ai389)
• [How do you handle GPU scheduling on Kubernetes for AI workloads?](/knowledge/ai361)
• [What is the best way to cache embeddings at scale?](/knowledge/ai419)
• [How do you handle model rollbacks safely in production?](/knowledge/ai429)
• [The 10 Best LLM Inference Servers in 2027](/knowledge/ai342)