How do you load-test an LLM inference service?

What load-testing an inference service actually means

Load-testing a traditional web service is mostly a question of requests per second. You fire N requests, count how many come back inside your latency budget, and find the knee. An LLM inference service breaks that model in three ways, and every mistake teams make downstream traces back to not internalizing these differences early.

First, work per request is variable and unknown at admission time. A request with a 200-token prompt and a 50-token completion costs a fraction of a request with a 30,000-token prompt and a 2,000-token completion. The prefill phase — processing the input prompt — is compute-bound and scales roughly with input length. The decode phase — generating output one token at a time — is memory-bandwidth-bound and scales with output length. A load test that sends 1,000 identical short prompts tells you almost nothing about a service whose real traffic includes long documents. Your virtual users are not interchangeable; their token shapes are the actual load variable.

Second, the server batches your requests together. Modern inference servers use continuous batching (sometimes called in-flight or iteration-level batching): rather than waiting for a batch to fill and finish, the scheduler admits new sequences into the running batch at each decode step and evicts finished ones. This means throughput and latency move in opposite directions as concurrency rises. At low concurrency, the GPU is idle between steps and per-request latency is excellent but tokens per second across the service is poor. As concurrency climbs, aggregate throughput rises steeply — until the batch saturates memory or compute, at which point per-request latency degrades sharply while total throughput plateaus. There is a genuine knee in this curve, and finding it is the entire point of the exercise.

Third, a single latency number is meaningless. Users experience streaming responses. The metric that governs perceived responsiveness is time-to-first-token (TTFT) — how long before text starts appearing. The metric that governs whether the response feels fast once it starts is inter-token latency (ITL), also expressed as time-per-output-token (TPOT). End-to-end latency is just TTFT plus ITL times output length, and reporting it alone hides which half is broken. A service with 300ms TTFT and 25ms ITL feels snappy. A service with 4s TTFT and 8ms ITL feels broken, even though its total latency on a 500-token response is lower.

So the working definition: a load test on an inference service is a controlled experiment that maps a realistic distribution of token shapes at a series of arrival rates onto the four output metrics — TTFT, ITL, output tokens per second per request, and total output tokens per second across the service — plus the error and rejection rates at each step. Everything else in this page is machinery for running that experiment honestly.

The adjacent workloads matter too. Embedding services, rerankers, and classification endpoints have far simpler load profiles — fixed input length, no decode phase, no streaming — and behave much more like classical HTTP services. If your platform serves both, do not reuse one methodology for the other. A RAG stack in particular has three separate load surfaces: the embedding call, the vector search, and the generation call, and the generation call is usually the only one that behaves nonlinearly. Test them separately before testing them together, or you will spend a day chasing a "latency spike" that turns out to be a vector index rebuild.

The step-by-step process

Run the test in phases. Skipping straight to "hammer it with 500 users" produces numbers you cannot defend in a capacity review.

Phase 1 — Fix the environment and write it down. Record model name and precision, server and version, GPU type and count, tensor-parallel degree, max model length, max batched tokens, KV cache memory fraction, and any speculative decoding or prefix caching settings. Two runs whose configs differ in any of these are not comparable. Put this block at the top of every result file. If you are testing behind a gateway, proxy, or autoscaler, record whether it is in path — a load balancer with a 30s idle timeout will silently kill long generations and show up as a mysterious error class.

Phase 2 — Build a realistic prompt corpus. Sample a few hundred to a few thousand real prompts from production logs if you have them and are permitted to use them; otherwise construct a synthetic corpus that matches the observed input-length and output-length distributions. The important thing is the distribution, not the median. Capture the long tail: if 5% of your traffic is 20k-token document summarization, your corpus must contain that 5%. Pin max_tokens per request from the corpus rather than a single global value, and if you want deterministic output lengths for a clean throughput measurement, disable early stopping and set ignore_eos where the server supports it. Also decide on temperature and seed: for capacity testing, greedy decoding with a fixed seed removes one source of run-to-run variance.

Phase 3 — Warm up and discard. The first requests after a server start pay for weight loading into GPU memory, CUDA graph capture, kernel autotuning, and cold KV cache allocation. Run at least 60 seconds of warm-up traffic and throw those measurements away. If prefix caching is enabled, warm-up also populates shared prefixes, which will make subsequent runs faster in a way that may or may not reflect production — decide deliberately which behavior you are measuring and note it.

Phase 4 — Establish the single-stream baseline. One request at a time, no concurrency. This gives you the floor: best-case TTFT and best-case ITL with zero queueing. Every later number is interpreted relative to this. If single-stream ITL is 30ms, then a batched run showing 45ms ITL means the batch is costing you 50% on per-token speed in exchange for whatever throughput multiple you gained.

Phase 5 — Step the load. Hold each step long enough to reach steady state — typically 2 to 5 minutes, longer if your output lengths are long — then step up. Two valid load models exist and they answer different questions. Closed-loop (fixed number of concurrent clients, each sending a new request as soon as the last finishes) answers "how does the service behave with N simultaneous users." Open-loop (fixed arrival rate in requests per second, independent of how fast the server responds) answers "what happens when demand exceeds capacity," and is the only model that will surface queue growth and coordinated omission honestly. Run both. Closed-loop finds the throughput knee; open-loop finds the collapse point.

Phase 6 — Correlate client metrics with server metrics. Client-side numbers tell you what users feel; server-side metrics tell you why. Scrape the inference server's own metrics endpoint during the run — running versus waiting request counts, KV cache utilization, preemption or swap counts, and batch size per iteration — alongside GPU utilization and memory from the device. The moment KV cache utilization pins near its ceiling and preemptions start, you have found your real capacity limit, regardless of what latency looks like for another 30 seconds.

Phase 7 — Push past the knee deliberately. Keep stepping until something breaks. You want to know the failure mode: does the server queue politely and degrade latency, does it start returning 429s, does it OOM, or does it silently drop connections? A service that fails by shedding load gracefully is operationally very different from one that falls over, and you cannot design the autoscaler or the retry policy without knowing which you have.

The tooling landscape and what each one is good for

There are two families of tool here and they are not substitutes.

Purpose-built inference benchmarks understand tokens. vLLM ships a benchmark script that measures TTFT, TPOT, and end-to-end latency directly and supports replaying sampled datasets at a chosen request rate, including Poisson arrival. NVIDIA's GenAI-Perf, part of the Triton ecosystem, does the same for OpenAI-compatible and Triton endpoints and reports token-level statistics natively. Hugging Face's text-generation-inference includes its own benchmarking harness. If your question is "what is this server's capacity for this model," start here — these tools already know that a response is a token stream and not a blob.

General HTTP load generators understand concurrency and scheduling, and know nothing about tokens until you teach them. Locust is Python-native, so parsing a streamed SSE response and emitting a custom TTFT metric is a dozen lines inside a task function; its master/worker mode distributes generation across machines and its web UI gives live charts. k6 scripts in JavaScript, integrates cleanly with CI pipelines, supports gRPC and WebSocket natively (useful for Triton or bidirectional streaming), and its k6-operator runs distributed tests inside Kubernetes. Vegeta and wrk2 are the right answer when you need a genuinely constant arrival rate — both were designed around open-loop, fixed-rate attack, which makes them well suited to the "what happens above capacity" question. Hey and oha are single-binary quick checks: excellent for "is the endpoint up and roughly how fast," useless for streaming. JMeter and Gatling bring enterprise reporting and GUI or DSL scenario building, which matters when the load test result has to go into a document someone else signs.

The practical pattern most teams land on: use the purpose-built benchmark for model and server capacity numbers, and use a general load generator for the full application path — gateway, auth, rate limiter, RAG retrieval, and the model together. Those two numbers will differ, and the gap between them is your infrastructure overhead, which is itself a finding worth having.

Whichever generator you pick, one constraint dominates: the load generator must not be the bottleneck. Streaming responses hold connections open for the whole generation, so a test at 200 concurrent streams holds 200 sockets. Python-based generators doing per-token parsing burn real CPU. Run the generator on separate hardware from the inference server, watch its own CPU during the run, and if the generator machine exceeds roughly 70-80% CPU, distribute across more nodes before believing any latency number it reports. Network placement matters too — a generator in a different region adds round-trip time straight into TTFT and will make a healthy server look sluggish.

Costs, timelines, and the numbers to expect

The direct cost of load-testing is GPU time on the target service plus the cheap CPU instances generating load. The expensive part is holding an accelerator idle-but-reserved while you iterate on the harness, so build and debug the harness against a small model on cheap hardware first, then point it at the real deployment.

Timeline. A first credible load test of a single model on a single deployment configuration is roughly a one-to-three-day exercise: half a day to assemble the prompt corpus and instrument TTFT/ITL correctly, half a day to validate the harness (including proving it can saturate the server at all), and one to two days of actual stepped runs plus analysis. Sweeping a configuration matrix — say four batch-size or KV-cache settings across two quantization levels — multiplies the run time, not the setup time, so budget the runs and automate them.

Run duration. Each concurrency step needs to reach steady state. With short outputs, 2 minutes per step is usually enough; with 1,000-token outputs and a slow ramp, 5 to 10 minutes per step is more honest. A full ramp of eight to ten steps therefore lands somewhere between 30 minutes and 2 hours per configuration. Add a soak run — the same load held for several hours — separately, because that is what catches memory leaks, cache fragmentation, and slow degradation that a 5-minute step will never show.

What good looks like, directionally. Avoid memorizing anyone's absolute numbers, including these; they are hostage to model size, precision, hardware generation, and sequence length. But the shapes are stable and worth knowing:

• Single-stream ITL on a well-served model is typically in the low tens of milliseconds, which reads as tens of tokens per second per user. Humans read at roughly 5-10 tokens per second, so per-user generation speed comfortably above that is usually adequate, and speed beyond it buys diminishing perceived value for chat — but matters enormously for agentic workloads where a model's output is consumed by another program with no patience.
• TTFT scales with prompt length because prefill is proportional to input tokens. Expect a long-context request to have TTFT an order of magnitude above a short one on the same server. If your product mixes both, a shared queue means long prefills will spike TTFT for short requests sitting behind them — this is chunked-prefill territory and is a real, tunable server setting on modern stacks.
• Aggregate throughput typically rises severalfold from concurrency 1 to the knee, because batching amortizes weight-loading memory traffic across sequences. The multiple is largest for small models and short sequences, and shrinks as KV cache pressure grows with long contexts.
• The knee is defined by KV cache capacity as often as by compute. Each concurrent sequence holds KV cache proportional to its current sequence length. Long contexts therefore reduce maximum concurrency dramatically — the same GPU that batches 100 short chats may batch only a handful of 30k-token requests.

Cost per token is a derived output of this test, not an input. Once you know sustainable output tokens per second at your latency SLO, divide your hourly instance cost by that rate to get cost per million output tokens. That single number is what makes the build-versus-buy comparison against a hosted API tractable, and it is why load-testing self-hosted inference is a finance exercise as much as an engineering one. Run the same test at a lower latency SLO and the cost per token rises, because you are buying headroom — quantifying that trade is often the most valuable artifact the whole exercise produces.

Where teams get it wrong

Coordinated omission. This is the single most common invalidating error. In a closed-loop test, when the server slows down, clients send fewer requests — so the slow period is under-sampled and your percentiles look far better than reality. The fix is an open-loop generator that maintains a fixed arrival rate regardless of response time, or a closed-loop tool that compensates by back-dating intended send times. If your load test reports p99 latency from a closed-loop run at saturation, that p99 is optimistic by a margin you cannot estimate.

Measuring end-to-end latency only. A tool that waits for the full response and reports one duration conflates prefill, queueing, and decode. You cannot tune anything from that number. Instrument TTFT separately, and derive ITL from timestamps on streamed chunks. Note the subtlety: providers often stream multiple tokens per SSE chunk, so chunk-arrival intervals are a lower bound on true per-token timing — record chunk timings and token counts both, and say which you are reporting.

Uniform prompts. Testing with one repeated prompt is the fastest way to produce a beautiful, useless graph. If prefix caching is on, a repeated prompt may be served almost entirely from cache and your prefill cost effectively disappears. Randomize prompts, and if you want to measure the un-cached path, deliberately vary the prefix.

No warm-up, or warm-up included in the numbers. Cold-start effects — weight load, graph capture, allocator warm-up — can dominate the first seconds. Including them inflates p99 and confuses everyone reading the report later.

Ignoring the failure mode. Teams find the knee, write it down, and stop. Then production hits 1.3x the knee during a spike and the service behaves in a way nobody predicted. Always run past the limit at least once and document exactly what happens: queue growth, 429s, timeouts at the gateway, preemption and recompute, or OOM.

Testing the model but not the path. The gateway, authentication, rate limiter, retrieval step, and any safety-classification hop all add latency and all have their own limits. A test that hits the inference server directly gives you model capacity; a test that hits the public endpoint gives you product capacity. Both are worth knowing and they are frequently far apart — it is not unusual for a retrieval step to add more latency than the generation itself for short answers.

Letting retries hide the failure. If your client library retries on timeout, a saturated server receives more load precisely when it is least able to serve it, and your success-rate graph looks fine while latency detonates. Disable retries in the load generator, or count them explicitly as a separate metric.

Comparing incomparable runs. Changing quantization, max batched tokens, tensor-parallel degree, or even the server version between runs and then comparing headline throughput is the classic way to draw a wrong conclusion. Change one variable per run. Keep the config block with every result.

Optimizing for the wrong metric. Maximum aggregate throughput and minimum per-user latency are in direct tension under batching. A batch-scheduling change that doubles total tokens per second may push per-user ITL from 25ms to 60ms. Which is correct depends entirely on the workload: an offline document-processing pipeline should be tuned to the throughput ceiling; an interactive assistant should be tuned to the latency SLO and accept lower utilization. Decide the objective before the test, not after seeing the graph.

Forgetting the autoscaler. If replicas scale on a metric, the load test is also a test of the scaling policy. Model loading takes real time — often minutes for large models — so a scale-up triggered at saturation arrives long after the spike that triggered it. Test the ramp rate your traffic actually exhibits, and measure how long the service stays degraded while new replicas warm.

Decision framework: choosing an approach

The right test depends on the question, and the question depends on where you sit in the deployment lifecycle.

"Which serving stack or configuration should we run?" — This is a capacity benchmark. Use a purpose-built token-aware benchmark, hit the inference server directly with no gateway in path, use a fixed synthetic input/output length distribution for comparability, and sweep the configuration matrix. Report throughput-versus-latency curves, never a single number.

"Can we handle Monday's launch?" — This is a capacity-planning test. Use a general load generator against the real public endpoint with the real corpus, in open-loop mode at your forecast peak arrival rate, with the autoscaler live. The output is pass/fail against your SLO plus the observed headroom.

"Did this change regress performance?" — This is a CI test. Short, fixed, cheap: a small fixed corpus at a fixed concurrency, run on identical hardware every time, comparing against a stored baseline with a tolerance band. k6 or a scripted benchmark invocation fits here. Keep it under a few minutes or it will be disabled within a month.

"Will it survive a week?" — This is a soak test. Hold roughly 70% of the measured knee for hours and watch for drift in latency, growth in memory, cache fragmentation, and slow leaks. Nothing about a 5-minute run predicts this.

"What breaks first, and how?" — This is a stress test. Open-loop, ramp well past capacity, and document the failure mode and the recovery time after load is removed. Recovery matters: a service that takes 20 minutes to drain a backlog after a 2-minute spike has an operational problem that peak-capacity numbers never reveal.

For tool selection specifically: choose a token-aware benchmark when the answer must be in tokens per second; choose Locust when you need custom per-request logic in Python and live visibility; choose k6 when the test lives in CI or the endpoint is gRPC or WebSocket; choose Vegeta or wrk2 when constant arrival rate is the whole point; choose a single-binary tool only for smoke checks.

Reading the results and turning them into decisions

A finished load test produces one artifact: a throughput-versus-latency curve annotated with the configuration that produced it. Plot aggregate output tokens per second on one axis and p95 TTFT or p95 ITL on the other, with each point labeled by concurrency. The knee is visually obvious. Draw a horizontal line at your latency SLO; where it crosses the curve is your usable capacity, and everything to the right is capacity you own but cannot sell.

From that curve, three decisions follow directly. Replica sizing: divide forecast peak demand by usable capacity per replica, then add headroom for the autoscaler's warm-up lag. Admission control: set your rate limit or queue depth so the service refuses work before it enters the degraded region rather than after — queueing past the knee converts a throughput problem into a latency catastrophe for every user simultaneously. Optimization priority: if TTFT is the constraint, you are prefill-bound and should look at chunked prefill, prefix caching, shorter prompts, or more compute; if ITL is the constraint, you are decode-bound and should look at quantization, speculative decoding, better memory bandwidth, or smaller batch sizes.

The result also feeds the build-versus-buy conversation. Cost per million output tokens at your SLO, computed from this curve, is directly comparable to a hosted provider's list price — with the caveat that self-hosting adds engineering time, on-call burden, and the utilization risk of paying for reserved accelerators during troughs. A service running at 30% average utilization has a real cost per token roughly three times its peak-rate figure, and that gap is the honest argument for elastic hosted capacity for spiky workloads.

Finally, the load test should not be a one-time event. Model versions change, prompt templates grow, retrieval contexts get longer, and traffic mixes drift toward whatever feature shipped last quarter. Re-run the standard test whenever the model, server version, or hardware changes, and keep the historical curves — the trend across versions is often more informative than any single run, and it catches the slow creep in prompt length that quietly halves your capacity over six months.

Related questions

What is the difference between TTFT and ITL?

TTFT is time-to-first-token: how long from request submission until the first output token arrives, dominated by queueing plus prefill of the input prompt. ITL is inter-token latency: the gap between subsequent tokens during decode. TTFT governs perceived responsiveness; ITL governs reading speed.

Should I use open-loop or closed-loop load generation?

Both, for different questions. Closed-loop (fixed concurrent users) maps the throughput-versus-latency curve and finds the knee. Open-loop (fixed arrival rate) reveals queue growth and avoids coordinated omission, so it is the only honest way to measure behavior above capacity.

How long should each load step run?

Long enough to reach steady state — usually 2 to 5 minutes, longer with long output lengths or slow autoscaling. Discard at least the first 60 seconds as warm-up. Short steps measure the transient, not the sustained capacity you actually need to plan against.

Does prefix caching invalidate my load test?

It changes what you are measuring. Repeated or shared-prefix prompts served from cache skip most prefill, producing optimistic TTFT. Decide deliberately: test with realistic prefix overlap if production has it, or deliberately vary prefixes to measure the cold path.

Can I load-test embedding and reranking endpoints the same way?

Partly. They have fixed input, no decode phase, and no streaming, so they behave much more like classical HTTP services — standard RPS and p95 latency suffice. Test them separately from generation before combining them into a full RAG path test.

FAQ

What metrics should a load test of an inference service report?

At minimum: time-to-first-token percentiles (p50/p90/p95/p99), inter-token latency percentiles, output tokens per second per request, aggregate output tokens per second across the service, request throughput, error rate broken out by class, and the concurrency or arrival rate at which each was measured. Alongside those, capture server-side KV cache utilization, running versus waiting request counts, preemption counts, and GPU utilization, so client symptoms can be traced to server causes.

Do I need a GPU to build the load-testing harness?

No. Build and debug the harness against a small model on CPU or a cheap accelerator, or even against a mock endpoint that streams synthetic tokens at a fixed rate. Validating that your generator correctly parses streamed chunks, records TTFT, and can itself sustain the target concurrency is entirely independent of the model. Only point it at expensive hardware once you trust the instrumentation.

How many concurrent users should I simulate?

Do not pick a number — sweep a range. Start at single-stream, then step through concurrency levels that roughly double each time until latency breaches your SLO or errors appear. The useful output is the whole curve, not a pass/fail at one arbitrary point. If you have production data, make sure your sweep brackets both current peak and forecast peak.

Why does throughput go up while my per-user latency gets worse?

Continuous batching. The server processes many sequences in the same decode step, amortizing the cost of streaming model weights from memory across all of them. Total tokens per second rises, but each individual sequence advances one token per step and the steps get slower as the batch grows. This trade-off is inherent to batched inference, not a bug, and choosing where on it to sit is a product decision.

Is it safe to load-test a production endpoint?

Generally no, not at saturation. Use a staging deployment on identical hardware for anything that pushes past capacity. If you must exercise production, stay well below the known knee, run during a low-traffic window, coordinate with whoever owns the on-call rotation, and have a kill switch on the generator. Load-testing a shared multi-tenant endpoint you do not own is a different matter entirely — check the provider's terms first, because sustained synthetic load usually violates them.

How often should this test be re-run?

Re-run the full sweep whenever the model, quantization, serving stack version, or hardware changes, and at least quarterly otherwise to catch drift in prompt lengths and traffic mix. Run a short CI-scale version on every significant change to the serving path so regressions surface within hours rather than at the next incident.

Sources

• vLLM documentation
• NVIDIA GenAI-Perf documentation
• Hugging Face Text Generation Inference
• Locust documentation
• k6 documentation
• Vegeta on GitHub
• wrk2 on GitHub
• Apache JMeter
• Gatling documentation
• Coordinated omission explained by Gil Tene

Related on PULSE

• [What causes high latency in LLM inference and how do you fix it?](/knowledge/ai389)
• [How do you scale LLM inference to handle thousands of concurrent users?](/knowledge/ai347)
• [What is the difference between vLLM, TGI, and Triton for LLM inference?](/knowledge/ai345)
• [The 10 Best LLM Inference Servers in 2027](/knowledge/ai342)
• [The 10 Best LLM Quantization and Inference Optimization Tools in 2027](/knowledge/ai388)
• [How do you choose an inference accelerator: GPU, TPU, or custom silicon?](/knowledge/ai415)