How do you reduce GPU costs when serving large language models?

The outcome you should expect

The number that matters is not GPU price per hour. It is cost per million tokens served at your latency target. Teams that fixate on hourly rates end up renting cheaper cards that serve fewer requests, and their unit economics get worse while their invoice looks better for one month. The correct framing is: how many tokens per second can one accelerator sustain while still meeting your time-to-first-token and inter-token-latency service levels, and what does that accelerator cost per hour?

When you start from a naive baseline — a Hugging Face Transformers generate() loop behind a Flask or FastAPI endpoint, one request at a time, no batching, no cache management — you are typically leaving somewhere between 70% and 90% of the GPU idle. The tensor cores sit unused while the process waits for the next token, memory is allocated conservatively so a large fraction of VRAM is reserved but unused, and every request recomputes the full prompt from scratch. That baseline is the reason "LLM serving is expensive" became conventional wisdom. It is expensive the way a delivery truck is expensive when you send it out with one parcel.

Realistic outcome bands, assuming you start from that naive baseline and apply changes in sequence:

• Switch to a modern inference engine with continuous batching and paged KV cache (vLLM, SGLang, TensorRT-LLM, TGI): 2-4x throughput at the same latency. This is the single largest jump and it requires no model changes at all — same weights, different runtime.
• Add weight quantization to 4-bit or 8-bit: 2-4x reduction in weight memory, which often means the same model fits on half or a quarter as many GPUs. A 70B-class model at fp16 needs roughly 140 GB just for weights and cannot fit on one 80 GB card; at 4-bit it needs roughly 35-40 GB and fits comfortably with room for KV cache.
• Add prefix/prompt caching where your traffic has shared prefixes: 20-60% compute reduction depending on how much of your average request is a repeated system prompt, few-shot block, or conversation history.
• Right-size the model — distillation, or simply testing whether a 7-8B model passes your evals instead of a 70B: often the largest single win of all, 5-10x, and the one teams skip because it feels like a step backwards.

Stack those and 3-5x total cost reduction is a conservative expectation. Some workloads — long shared system prompts, high concurrency, tolerant latency budgets — do considerably better. What you should *not* expect is that all four multiply cleanly. They contend for the same resource. Quantization frees memory that continuous batching immediately consumes with a larger batch; prefix caching frees compute that then exposes a memory-bandwidth ceiling. Measure the stack, not the parts.

There is a second outcome worth naming: variance collapse. A well-tuned serving stack does not just get cheaper, it gets *predictable*. Naive serving has terrible tail latency because a long request blocks the queue behind it. Continuous batching decouples requests from each other, so your p99 stops being a function of whoever happened to ask for a 4,000-token essay. Predictable latency is what lets you run at 70-80% utilization instead of over-provisioning to 30% for headroom — and that over-provisioning is frequently a larger line item than any inefficiency inside the engine itself.

What drives that outcome

Three physical constraints govern everything. Understand which one binds your workload and the optimization order becomes obvious rather than a menu of tricks.

Memory capacity. Model weights plus KV cache plus activations must fit in VRAM. Weights are fixed per model and precision: roughly 2 bytes per parameter at fp16, 1 byte at int8/fp8, ~0.5-0.6 bytes at 4-bit including quantization overhead like scales and zero-points. KV cache grows with batch size × sequence length × layers × heads × head dimension × 2 (keys and values) × bytes per element. For long contexts and high concurrency, KV cache routinely exceeds the weights. This is why paged attention mattered so much: classical implementations reserved a contiguous worst-case block per sequence, so a request that *might* generate 4,000 tokens reserved memory for 4,000 tokens even if it stopped at 40. Paging that cache into fixed-size blocks, allocated on demand, recovers most of the waste and lets you admit far more concurrent sequences.

Memory bandwidth. Autoregressive decoding is bandwidth-bound, not compute-bound. Generating one token requires reading every weight in the model from HBM. At batch size 1, you read the entire model per token and the arithmetic intensity is dismal — you are using a tiny fraction of available FLOPs. Batching amortizes that read across many sequences, which is precisely why batching is the highest-leverage change. It is also why quantization speeds up decoding even when the math is done in higher precision: fewer bytes to read per token.

Compute. Prefill — processing the input prompt — is compute-bound and parallel across all prompt tokens. Decode is bandwidth-bound and serial. These two phases have opposite hardware appetites, which is why mixing them in one batch causes interference: a long prefill stalls the decode steps sharing that batch, spiking inter-token latency for everyone. Chunked prefill and prefill/decode disaggregation exist to solve exactly this.

The diagram above is the whole cost model in one loop. Every optimization is an intervention at one of those nodes. Quantization shrinks the weight footprint so more blocks fit. Prefix caching short-circuits the prefill branch. Continuous batching keeps the scheduler node saturated. Speculative decoding makes each decode step produce more than one token. Hardware selection changes how fast the bandwidth-bound edge runs. If you cannot say which node your bottleneck sits on, you are guessing — and guessing at this layer is how teams spend a quarter migrating to a framework that addresses a constraint they did not have.

One adjacent effect worth flagging: the same reasoning transfers directly to embedding and reranking services, which most RAG stacks run alongside the generator. Embedding workloads are almost purely prefill — no autoregressive decode at all — so they are compute-bound and batch beautifully. Teams that co-locate an embedding model on the same GPU as a chat model usually regret it, because the embedding batch's compute burst wrecks the chat model's inter-token latency. Separate them onto different cards, or at minimum different processes with hard memory partitions. The same lesson applies to fine-tuning jobs sharing inference hardware to "use idle capacity": the idle capacity is your latency headroom.

Benchmarks and realistic ranges

Be skeptical of any single throughput number, including ones you generate yourself, unless it is stated with the full configuration: model, precision, input length, output length, concurrency, and the latency constraint it was measured under. A throughput figure without a latency ceiling is meaningless — you can always get more tokens per second by making everyone wait longer.

How to construct a benchmark that is actually useful. Replay your own traffic, not a synthetic uniform load. Capture a few thousand real requests with their true input and output lengths, then replay them at increasing arrival rates. Plot tokens/second against p95 inter-token latency. The useful output is a curve, not a point: it tells you the maximum sustainable request rate at your SLO, and everything past the knee is throughput you cannot actually sell. Most published benchmarks report the far right of that curve, where latency has already blown past anything a chat UI could use.

Sizing arithmetic you can do on paper before renting anything. Weight memory ≈ parameters × bytes-per-parameter. KV cache per token ≈ 2 × layers × kv_heads × head_dim × bytes-per-element; multiply by context length and concurrent sequences. Models using grouped-query or multi-query attention have dramatically smaller KV caches than older multi-head architectures — often 4-8x smaller — which changes concurrency math more than any serving flag. Check the architecture's KV head count before you assume a model is expensive to serve at high concurrency. Then: available KV memory = total VRAM − weights − activations − runtime overhead (budget 2-4 GB), and concurrency ≈ available KV memory ÷ per-sequence KV footprint. That single calculation predicts most of your cost structure and takes five minutes.

Precision trade-offs, stated honestly. 8-bit weight quantization (int8 or fp8 on hardware that supports it natively) is close to free in quality terms for most tasks and is the safe default. 4-bit methods — AWQ, GPTQ, and their descendants — typically retain quality well on general benchmarks but degrade unevenly: long-context reasoning, code generation, structured output adherence, and non-English performance tend to suffer first and are underrepresented in standard eval suites. Run your own task-specific evals before and after. The honest statement is that 4-bit *usually* works and *sometimes* quietly does not, and the failure is invisible unless you look for it in the dimension that matters to your product. KV cache quantization is a separate lever: quantizing the cache to 8-bit roughly doubles the concurrency ceiling for long-context workloads and is frequently a better trade than quantizing weights further.

Prefix caching yields depend entirely on traffic shape. If your average request is a 2,000-token system prompt plus a 50-token user turn, the cacheable fraction of prefill is enormous and hit rates above 80% are achievable. If every request is a unique document dropped into the context window, prefix caching does approximately nothing and you should not spend engineering time on it. Compute your cacheable fraction from real logs first: take the median shared-prefix length divided by median total input length. That ratio is your ceiling.

Speculative decoding — a small draft model proposes several tokens, the target model verifies them in one parallel forward pass — delivers real speedups because it converts bandwidth-bound decode steps into compute-bound verification steps. The gain scales with the acceptance rate, which depends on how well the draft model matches the target's distribution on *your* domain. Generic draft models on specialized domains accept poorly, and a low acceptance rate can make things slower than no speculation at all, since you pay for rejected drafts. It is a strong lever at low-to-moderate load, where GPUs are bandwidth-starved anyway. Under heavy batching it helps less, because large batches already provide the arithmetic intensity speculation was manufacturing.

Hardware. Newer datacenter accelerators generally win on cost-per-token despite higher hourly rates, because throughput scales faster than price — but only if your workload can saturate them. A card with double the bandwidth serves you nothing at batch size 2. Older or smaller cards can be genuinely better for small quantized models under modest load, and for embedding or reranking work. Alternative accelerators with very large memory pools are attractive for models that would otherwise require multi-GPU tensor parallelism, because collapsing a two-card deployment to one card eliminates interconnect overhead and halves the failure surface, not just the price. Always price the *whole node*: interconnect, host memory, and egress often exceed the GPU line item for retrieval-heavy services.

Finally, benchmark the idle. Utilization is frequently the dominant cost. A cluster averaging 20% utilization is paying a 5x tax that no framework choice will recover. Autoscaling on queue depth, scale-to-zero for low-traffic models, spot or preemptible capacity for interruptible batch work, and consolidating several small models onto one card via multi-model serving all attack that tax directly. For genuinely spiky traffic, per-token serverless endpoints can undercut self-managed GPUs simply by charging nothing when nobody is asking.

Risks, edge cases, and failure modes

Silent quality regression. The most common failure is shipping an aggressive quantization and discovering three weeks later that structured output adherence dropped, JSON parsing errors climbed, or a non-English segment degraded. Gate every precision change behind a task-specific eval set of at least a few hundred real examples, and keep a small fp16 canary deployment so you can A/B against the unquantized model on live traffic.

Preemption thrash. Continuous batching schedulers admit requests optimistically. When memory pressure spikes — several very long generations at once — the scheduler preempts sequences, discarding or swapping their KV cache. Recomputation then burns the compute you saved. Symptom: throughput that collapses non-linearly past a certain concurrency instead of plateauing. Fix by capping max sequence length, reserving a memory margin rather than pushing GPU memory utilization to the ceiling, and admission-controlling on estimated total tokens rather than request count.

Prefix cache correctness and privacy. Sharing cached KV blocks across requests is safe when the prefix is genuinely identical. It becomes a data-leakage vector if cache keys are computed carelessly — a hash collision, or a cache scoped across tenants when prefixes include user-specific content. Scope caches per tenant unless the shared prefix is provably public, and treat cache keying as a security-review item, not a performance detail.

Compilation and version lock-in. Engines that compile models to hardware-specific artifacts deliver excellent latency but bind you to an exact GPU model, driver, and runtime version. A cloud provider capacity shift or a driver upgrade can invalidate your artifacts and force a rebuild that takes hours. Keep a non-compiled fallback path that can serve degraded-but-working traffic, and store build recipes in CI so a rebuild is a pipeline run rather than an archaeology project.

Spot instance interruption. Preemptible capacity is 60-90% cheaper and genuinely appropriate for batch inference, evals, and offline enrichment. It is not appropriate for interactive traffic without a warm on-demand fallback, because model loading is slow — tens of seconds to minutes for large weights — and a two-minute preemption warning is not enough to cold-start a replacement. Mixed fleets work: on-demand baseline sized to p50 traffic, spot for the peak.

The multi-GPU cliff. Tensor parallelism across cards adds all-reduce communication on every layer. Within one node over a high-speed interconnect that cost is modest. Across nodes over standard networking it is severe, and a model split across two nodes can be slower than the same model quantized onto one. Before scaling out, exhaust the options that let you scale *in*: quantization, KV cache quantization, a smaller model, shorter max context.

Long-context blowup. KV cache scales linearly with context length, so a 128K-context request can consume as much memory as dozens of ordinary ones. A handful of them will destroy concurrency for everyone else. Segregate long-context traffic onto a separate pool with its own capacity rather than letting it contend with chat traffic.

Over-optimizing the wrong layer. The costliest failure mode is spending six weeks on kernel-level tuning when the actual problem is that 40% of requests should never have hit the large model. Routing simple requests to a small model, caching identical responses at the application layer, trimming bloated system prompts, and capping max_tokens are unglamorous and frequently outperform everything in this article combined. Prompt length is a cost input: a 3,000-token system prompt that could be 600 tokens is a permanent 5x tax on every prefill you serve.

A practical rollout plan

Sequence matters. Each step should be independently measurable and independently revertible, and you should not start the next one until the previous is proven in production.

Week 0 — instrument. You cannot optimize what you cannot see. Capture per-request input tokens, output tokens, queue wait, time-to-first-token, inter-token latency, and GPU memory and utilization. Compute your current cost per million tokens. Log a representative traffic sample for replay. Skipping this step is why most optimization efforts cannot prove they worked.

Week 1 — engine swap. Move to a serving engine with continuous batching and paged KV cache, keeping the model, precision, and hardware identical so the comparison is clean. Replay your captured traffic against both stacks. Expect the largest single improvement here.

Week 2 — memory tuning. Set max model length to the true p99 of your traffic rather than the architecture's maximum. Tune the memory utilization fraction upward until you see preemption, then back off. Enable prefix caching if your cacheable fraction justifies it.

Week 3 — precision. Try 8-bit first, evaluate, then consider 4-bit if memory is still the binding constraint. Gate on task-specific evals, not general benchmarks. Consider KV cache quantization separately — for long-context workloads it often beats further weight quantization.

Week 4 — model right-sizing. Run your evals against smaller models in the same family. If a 7-8B model passes, that is a larger win than every runtime optimization stacked together. Distillation is the deliberate version of this and is worth the training cost when your task is narrow and volume is high.

Week 5 — fleet economics. Autoscale on queue depth. Move batch work to preemptible capacity. Consolidate low-traffic models. Re-evaluate hardware now that you know your real bottleneck. Only now does the "which GPU" question have a defensible answer.

Two organizational notes. First, assign the cost metric an owner. Cost per million tokens drifts upward silently as prompts grow, context windows expand, and new features add retrieval chunks. Without a named owner and a dashboard, you will re-run this whole exercise in nine months. Second, put a regression gate in CI: replay a fixed traffic sample against every serving-config change and fail the build if cost per million tokens or p95 latency regresses beyond a threshold. Serving configuration is production infrastructure and deserves the same discipline as application code.

Related questions

Is it cheaper to use an API provider than to self-host?

Below roughly a few hundred million tokens per month, hosted APIs usually win — you pay nothing for idle capacity, and no engineer maintains the stack. Self-hosting wins at sustained high volume, with strict data-residency requirements, or with heavy fine-tuning. Compute both at your real utilization, not peak.

Does a smaller model always cost less to serve?

Usually, but not proportionally. A model one-tenth the size is bandwidth-bound differently and may not deliver ten times the throughput, especially at low batch sizes where fixed overheads dominate. The reliable win comes from fitting on fewer, cheaper cards and enabling much higher concurrency.

How much does context length actually cost?

Prefill scales roughly quadratically with prompt length for attention and linearly for the rest; KV cache memory scales linearly. Doubling context more than doubles prefill cost and halves your concurrency ceiling. Trimming a bloated system prompt is one of the cheapest optimizations available.

Should batch and interactive traffic share GPUs?

No. Batch prefill bursts wreck interactive inter-token latency. Run separate pools with separate scaling policies, put batch work on preemptible capacity, and let interactive traffic keep the on-demand baseline. The isolation costs a little capacity and saves your tail latency.

Does fine-tuning reduce serving cost?

Indirectly and often substantially. A fine-tuned small model can replace a large model plus a long few-shot prompt, cutting both parameter count and prompt length simultaneously. Serving many adapters on one base model amortizes hardware across tasks that would otherwise each need their own deployment.

FAQ

What is the single highest-leverage change for most teams?

Moving from a naive generation loop to an inference engine with continuous batching and a paged KV cache. It requires no model change, no retraining, and no quality trade-off — the same weights simply serve far more concurrent requests. Teams routinely see multiple-fold throughput improvements from this alone, and everything else in the stack builds on it.

How do I know whether I am memory-bound or bandwidth-bound?

Watch GPU memory utilization and streaming-multiprocessor utilization together while increasing concurrency. If memory fills and the scheduler starts preempting before compute utilization rises, you are memory-bound — quantize weights or the KV cache. If memory has headroom but throughput plateaus while utilization stays moderate, you are bandwidth-bound — batch harder or consider speculative decoding.

Is 4-bit quantization safe for production?

Often, but never assume it. Modern 4-bit methods hold up well on general benchmarks while degrading unevenly on long-context reasoning, code, structured output, and non-English text. Validate on a task-specific eval set of real examples, keep an unquantized canary for A/B comparison, and treat 8-bit as the conservative default when quality risk is high.

Can these techniques be combined, or do they conflict?

They combine, but gains do not multiply cleanly because they compete for the same resources. Quantization frees memory that batching immediately consumes; prefix caching frees compute that exposes a bandwidth ceiling. Apply them one at a time, measure after each, and expect the stack to land meaningfully below the product of the individual claims.

What should I do about idle GPU cost?

Treat utilization as a first-class metric. Autoscale on queue depth rather than CPU, scale low-traffic models to zero, consolidate several small models onto one accelerator, and move interruptible batch work to preemptible capacity. For genuinely spiky traffic, per-token serverless endpoints eliminate idle cost entirely and frequently beat self-managed GPUs on total spend.

How often should I revisit these decisions?

Quarterly at minimum, and whenever a new model generation or accelerator lands. This area moves quickly — architectural changes like grouped-query attention shifted the concurrency math more than any serving flag. Keep the traffic replay harness and the cost dashboard permanently in place so re-evaluating is a one-day exercise rather than a project.

Sources

• vLLM documentation
• vLLM project repository
• NVIDIA TensorRT-LLM
• Hugging Face Text Generation Inference
• SGLang project repository
• FlashAttention repository
• AWQ: Activation-aware Weight Quantization
• Hugging Face quantization guide
• NVIDIA developer blog
• AWS EC2 on-demand pricing

Related on PULSE

• [What is model serving and how is it different from a REST API?](/knowledge/ai381)
• [The 10 Best Model Serving Frameworks in 2027](/knowledge/ai372)
• [The 10 Best Embedding Models for Search and RAG in 2027](/knowledge/ai362)
• [How do you version datasets and models for reproducibility?](/knowledge/ai383)
• [How do you deploy AI models at the edge?](/knowledge/ai399)