What causes high latency in LLM inference and how do you fix it?

What it is and why it matters

Latency in LLM serving is not one number. It splits into two measurements that behave differently and have different causes, and conflating them is the single most common reason optimization work fails to move the metric anyone actually cares about.

Time-to-first-token (TTFT) covers everything from request arrival to the first streamed character: queue wait, tokenization, and the *prefill* pass where the model processes the entire prompt in parallel. Prefill is compute-bound — it is a large batched matrix multiplication over all prompt tokens at once, so it scales with prompt length and GPU FLOPs. A 500-token prompt and a 50,000-token prompt have wildly different TTFT on identical hardware.

Inter-token latency (ITL), sometimes called time-per-output-token, covers each subsequent token. Decode is *memory-bandwidth-bound*: to produce one token, the GPU must stream the entire set of model weights out of HBM and into compute units, plus read the accumulated KV-cache for every prior token in the sequence. Arithmetic intensity collapses — you do a tiny amount of math per byte moved. This is why a model that fits in VRAM with room to spare can still feel slow: the bottleneck is data movement, not the tensor cores, which sit largely idle during decode.

Do the back-of-envelope math and the picture clarifies. A 70B-parameter model in FP16 is roughly 140 GB of weights. On a GPU with about 2 TB/s of memory bandwidth, streaming those weights once takes on the order of 70 ms — which sets a hard floor of roughly 14 tokens/second for single-stream batch-size-1 decode before you account for the KV-cache, attention, or any overhead. Quantize the same weights to INT4 and you move ~35 GB per token instead, roughly quartering that floor. That single calculation explains why quantization is usually the highest-leverage latency fix available, and why chasing a faster kernel while running FP16 weights on a bandwidth-starved card is wasted effort.

The KV-cache is the other half of the memory story and the part teams underestimate. For every token in every active sequence, the model stores a key and value vector per attention layer. Cache size scales linearly with batch size *and* with sequence length, so a service with 32 concurrent users at 8K context can be carrying tens of gigabytes of cache alongside the weights. When the cache grows past available VRAM, the serving framework starts preempting or swapping sequences, and tail latency explodes — the P99 goes from tolerable to unusable while the P50 barely moves.

Why this connects to revenue: perceived responsiveness governs whether an AI feature gets used at all. Interactive assistants, sales-copilot surfaces, and support deflection bots all show measurable abandonment as first-token delay stretches into multiple seconds. On the cost side, latency and throughput are the same coin — a serving stack that halves per-token latency typically also serves substantially more concurrent users per GPU, which directly reduces the GPU-hours behind every request. Inference efficiency is therefore both a conversion-rate lever and a gross-margin lever, which is exactly why it belongs on an ops roadmap rather than buried in an infrastructure backlog.

One more framing that saves time: batch size determines which regime you are in. At batch size 1 you are almost purely bandwidth-bound and quantization dominates. At large batch sizes the weight read is amortized across many sequences, arithmetic intensity climbs, and you become compute-bound — at which point quantization helps far less and kernel efficiency, attention implementation, and scheduling matter far more. Know your operating batch size before you pick a fix.

The step-by-step process

Optimization without measurement is guesswork. Work this sequence in order; each step tells you whether the next one is even relevant.

Step 1 — Instrument before you touch anything. Record TTFT, ITL, end-to-end latency, and throughput as distributions, not averages. Report P50, P95, and P99. Averages hide the queueing behavior that produces user-visible stalls. Capture these under a load pattern that mirrors production: real prompt-length distribution, real concurrency, real output-length distribution. A single-query benchmark on an idle GPU tells you almost nothing about a service handling bursty traffic.

Step 2 — Establish the roofline. Compute the theoretical floor: (bytes of weights) ÷ (GPU memory bandwidth) gives you the minimum seconds per decode step at batch size 1. Compare your measured ITL against it. If you are within roughly 2× of the floor, your serving stack is already reasonably efficient and further software tuning yields little — the remaining wins are quantization or different hardware. If you are 5–10× off the floor, you have a software problem worth chasing: bad batching, a naive attention implementation, per-token Python overhead, or CPU-GPU synchronization stalls.

Step 3 — Profile the split. Break the request into prefill time versus decode time. If TTFT dominates and prompts are long, your problem is attention and prefill compute — go to FlashAttention, prefix caching, and prompt trimming. If total time is dominated by decode across many output tokens, your problem is bandwidth — go to quantization and batching. Teams that skip this step routinely deploy a decode optimization against a prefill-bound workload and see no change.

Step 4 — Check memory headroom. Measure peak allocated memory during a realistic forward pass. If weights plus peak KV-cache exceed roughly 80–85% of VRAM, you will see erratic latency spikes from preemption, cache eviction, or offloading regardless of what else you tune. Fix capacity first: quantize, cap max context length, reduce max concurrent sequences, or add a GPU.

Step 5 — Adopt a real serving engine. Naive per-request generation loops leave enormous performance on the table. A production engine gives you continuous (in-flight) batching, paged KV-cache management, and optimized kernels together. This is usually the largest single-step improvement available to a team still serving from a hand-rolled loop.

Step 6 — Quantize. Weight-only INT4/INT8 schemes cut the bytes moved per decode step nearly proportionally. Validate accuracy on your actual task, not a generic leaderboard.

Step 7 — Attack the remaining bottleneck. Long context → FlashAttention. Repeated system prompts → prefix caching. Low batch, interactive workload → speculative decoding. Model too large for one GPU → tensor parallelism. High concurrency with idle GPU → tune scheduler and batch limits.

Step 8 — Re-measure and stop. Confirm the win under the same load pattern, then stop. Each additional layer adds operational complexity, and complexity has its own cost.

Costs, timelines, and typical ranges

Budget both the engineering time and the hardware, because the cheap fix is often the one nobody schedules.

Swapping to a production serving engine is usually a 1–3 day task for one engineer if the model is a standard architecture with existing support: pull the container, point it at the weights, expose an OpenAI-compatible endpoint, run a load test. Reported throughput gains over naive HuggingFace generation loops commonly land in the low single-digit multiples, driven mostly by continuous batching keeping the GPU busy between requests rather than idling while a batch drains. This is nearly always the best time-to-value step available and it costs nothing in hardware.

Weight quantization takes hours to a couple of days depending on the method. Post-training weight-only schemes like INT4 group-wise quantization run offline in minutes to a few hours on a single GPU for mid-size models, and the artifacts are often already published for popular open-weight models — meaning zero conversion time if you trust the source. Budget the real time for *evaluation*: build a task-specific eval set of at least a few hundred examples and compare quantized against full precision before shipping. Published accuracy deltas for careful 4-bit weight-only methods are typically small on general benchmarks, but degradation concentrates in multi-step reasoning, math, and long-form code — precisely the tasks where teams notice regressions after launch rather than before.

Attention kernel upgrades are usually a dependency bump plus a config flag, measured in hours, assuming your GPU generation is supported. Modern serving engines enable an optimized attention path by default, so many teams get this without doing anything explicit. The gain concentrates entirely in long-context prefill; on short prompts it is close to noise.

Prefix caching is a configuration flag in engines that support it, but the design work around it is where the time goes: you have to structure prompts so the shared portion — system instructions, tool schemas, few-shot examples, retrieved boilerplate — is a stable, byte-identical prefix, with the variable user content strictly appended. Teams that interleave dynamic content into the system prompt get near-zero hit rate and conclude the feature does not work. The memory cost is real: cached prefixes occupy KV-cache space that would otherwise hold active sequences, so it trades capacity for TTFT.

Speculative decoding is a multi-week effort if you need to train or tune draft heads, or a few days if a compatible draft model already exists. It improves ITL at low batch sizes by verifying several candidate tokens per forward pass, but it consumes extra compute for the verification step. At high batch sizes the GPU is already saturated and the technique stops paying — sometimes it makes things worse.

Tensor parallelism across multiple GPUs is days of work plus meaningful hardware spend. It is the right answer when a model genuinely cannot fit in one device's memory even quantized, because the alternative — CPU offloading — is catastrophically slow, often an order of magnitude worse per token. Its cost is inter-GPU communication on every layer, so it is only sane over a high-bandwidth interconnect; across ordinary PCIe or a network hop, communication overhead can eat the entire benefit.

Hardware. Newer datacenter GPUs bring both higher memory bandwidth and support for lower-precision formats like FP8 with native tensor-core acceleration. For memory-bound decode, bandwidth ratio is a good first-order predictor of speedup. But the ordering matters financially: a well-tuned software stack on last-generation hardware routinely beats an untuned stack on current-generation hardware, so buying capacity before fixing the serving path wastes capital. Optimize software, then size hardware to the measured requirement.

Where teams get it wrong

Benchmarking with a single request on an idle GPU. This measures the best case that no production user will ever experience. Real services queue. Load-test with concurrency at or above your expected peak and report P99, because that is the experience driving complaints.

Optimizing the wrong half. A team spends weeks on quantization for a RAG workload with 12,000-token prompts and 80-token answers. Almost all the wall-clock was prefill; the win was invisible. Profile the prefill/decode split first — it is a ten-minute measurement that redirects weeks of work.

Treating average latency as the metric. Continuous batching improves throughput and average latency while potentially *increasing* the worst-case wait for an unlucky request, because scheduling decisions favor aggregate GPU utilization. If your product promises a responsiveness guarantee, you need P99 and possibly per-request priority classes, not a better mean.

Ignoring context-length growth. A service benchmarked at 2K context gets deployed and users paste in documents. KV-cache scales linearly with sequence length, and attention cost scales worse than linearly, so the memory profile that fit comfortably in testing overflows in week two. Set and enforce a maximum context length, and load-test at that maximum rather than at the median.

Shipping quantization without a task-specific eval. General benchmark scores can look nearly unchanged while a specific downstream behavior — structured JSON output, tool-call argument fidelity, multi-step arithmetic — degrades noticeably. Always evaluate on the actual task with the actual prompts.

Leaving GPU utilization unmeasured. If utilization sits low under load, the bottleneck is not the model at all — it is the scheduler, the tokenizer, a synchronous preprocessing step, network serialization, or a Python-side loop blocking the request path. No amount of kernel tuning fixes a starved GPU. Check utilization before assuming a model-level problem.

Chaining every technique at once. Quantization plus speculative decoding plus aggressive parallelism plus a custom kernel yields a stack nobody can debug, where techniques interact badly — speculative decoding's benefit shrinks at the large batch sizes continuous batching creates, and quantized draft models can drop acceptance rates enough to erase the gain. Add one layer, measure, keep it only if it earns its complexity.

Underestimating the non-GPU path. Tokenization, request validation, retrieval calls, guardrail model passes, and response post-processing all sit on the critical path. A retrieval step adding a few hundred milliseconds can exceed everything you saved in the model. Trace the full request, not just the forward pass.

Forgetting cold starts. Loading a large model from disk into GPU memory takes minutes, and compiled engines take additional build time. Autoscaling that spins up replicas on demand will serve terrible latency during every scale-out event. Keep warm capacity or pre-provision.

Decision framework: when to choose what

Pick from the bottleneck, not from what is fashionable. The decision tree below encodes the ordering that keeps effort proportional to payoff.

The first branch is capacity. If the model plus expected peak KV-cache does not fit in VRAM with headroom, nothing else matters — fix that before tuning anything. Quantize first because it is cheapest; cap max context and max concurrent sequences second; add GPUs and tensor parallelism only if quantization cannot close the gap. Never accept CPU offloading as a steady-state answer for an interactive service.

The second branch is the prefill/decode split. Long prompts, short answers → optimize prefill: efficient attention, prefix caching for the stable portion, and prompt reduction (better retrieval beats more retrieved chunks). Short prompts, long answers → optimize decode: quantization and continuous batching.

The third branch is concurrency. High concurrency with many users → continuous batching and paged cache management deliver the biggest wins because they eliminate GPU idle time between requests. Low concurrency with strict interactive requirements → speculative decoding is worth the complexity precisely because there is spare compute to spend on verification.

The fourth branch is hardware generation. If your GPUs support native FP8 or similar low-precision formats with tensor-core acceleration, use them — you get the bandwidth reduction of quantization without the accuracy penalty typical of aggressive integer schemes. If not, weight-only INT4/INT8 remains the pragmatic path.

A pragmatic default for most teams: production serving engine with continuous batching and paged KV-cache, plus efficient attention, plus INT4 or FP8 weights if accuracy holds on your eval, plus prefix caching if your prompts share a stable header. That combination is achievable in about a week and captures the large majority of available improvement. Everything past it is diminishing returns you should only buy with a measured business reason.

Related questions

Does quantization always reduce latency?

No. It reduces bytes moved per decode step, so it helps most at small batch sizes where you are bandwidth-bound. At large batch sizes the workload shifts toward compute-bound, and if the quantized kernel dequantizes on the fly it can even be slower than a well-optimized FP16 path.

Why is my first request after deployment so slow?

Cold start. Loading tens of gigabytes of weights from storage into GPU memory takes minutes, and compiled inference engines add graph-build time. Keep replicas warm, pre-pull model artifacts into local storage, and avoid autoscaling policies that create replicas on demand for interactive traffic.

Does a longer context window slow down every token?

Yes, gradually. Attention over the KV-cache grows with sequence length, so per-token cost rises as generation proceeds and as the prompt lengthens. The prefill cost of a long prompt is paid once; the decode penalty is paid on every subsequent token.

Can I fix latency by using a smaller model?

Often the highest-leverage option. A smaller model reduces bytes moved per token proportionally to parameter count. If a distilled or smaller model meets your quality bar on a task-specific eval, it beats every serving optimization on both latency and cost.

Does streaming actually reduce latency?

It reduces *perceived* latency, not total time. Users judge responsiveness by first-token delay, so streaming makes a 4-second generation feel fast if the first token arrives in 300 ms. Optimize TTFT specifically if streaming is your interface.

FAQ

What is the single biggest cause of high latency in LLM inference?

Memory bandwidth. During decode, the GPU must read the full set of model weights plus the KV-cache from HBM for every single token generated, while performing relatively little arithmetic per byte moved. Compute units sit underused. That is why reducing bytes — through quantization, smaller models, or better cache management — beats almost every other intervention at low batch sizes.

How do I measure LLM latency correctly?

Measure TTFT and inter-token latency separately, under realistic concurrency, using GPU-side timing events rather than wall-clock calls that miss asynchronous execution. Warm up first to exclude compilation and allocation effects. Report P50, P95, and P99 across a few hundred requests with a production-like prompt-length distribution. Single-query numbers on an idle GPU are not predictive.

Should I optimize software or buy faster GPUs first?

Software first, essentially always. An untuned stack commonly runs several times slower than its hardware roofline, so a serving-engine swap plus quantization can deliver more improvement than a hardware generation jump — and costs nothing in capital. Once you are within roughly 2× of the bandwidth floor, further gains genuinely require more bandwidth, and hardware becomes the right purchase.

Why does P99 latency spike while average latency looks fine?

Almost always queueing and KV-cache pressure. When the cache fills, the scheduler preempts or evicts sequences, and those unlucky requests wait far longer than the median. Continuous batching optimizes aggregate throughput, which can worsen individual worst cases. Cap max sequence length and concurrent sequences, and add priority classes if some traffic needs a guarantee.

Does prefix caching help every application?

Only when requests share a byte-identical leading prefix — a stable system prompt, fixed tool schemas, or constant few-shot examples. Hit rates collapse if dynamic content like timestamps or user IDs is injected early in the prompt. Restructure so everything variable is appended last, and expect no benefit for workloads with fully unique prompts.

Is speculative decoding worth the complexity?

At low batch sizes with strict interactive requirements, yes — it produces multiple tokens per verification pass and meaningfully cuts inter-token latency. At high concurrency it stops paying, because the GPU is already saturated and the extra verification compute competes with real work. Treat it as a targeted fix for single-stream responsiveness, not a general throughput lever.

Sources

• vLLM documentation
• Efficient Memory Management for LLM Serving with PagedAttention (arXiv)
• FlashAttention: Fast and Memory-Efficient Exact Attention (arXiv)
• NVIDIA TensorRT-LLM
• Hugging Face Text Generation Inference documentation
• AWQ: Activation-aware Weight Quantization (arXiv)
• GPTQ: Accurate Post-Training Quantization for Generative Transformers (arXiv)
• Fast Inference from Transformers via Speculative Decoding (arXiv)
• NVIDIA Technical Blog
• PyTorch documentation: scaled_dot_product_attention

Related on PULSE

• [The 10 Best AI Tools for Shopping Cart Development in 2027](/knowledge/ai0245)
• [The 10 Best AI Tools for Favicon and Icon Design in 2027](/knowledge/ai0253)
• [The 10 Best AI Tools for UI Mockups in 2027](/knowledge/ai0251)
• [The 10 Best AI Tools for Landing Page Design in 2027](/knowledge/ai0248)
• [The 10 Best AI Tools for Product Page Design in 2027](/knowledge/ai0244)