What do AI infrastructure teams do differently when scaling LLM inference in 2027?

What it is and why it matters

"Scaling LLM inference" in 2027 no longer means the thing it meant in 2023 — adding replicas behind a load balancer until latency drops. It means running a fleet where every request has two structurally different phases, where memory bandwidth rather than compute is usually the binding constraint, and where the difference between a competent and an incompetent serving configuration is routinely 5–10x in cost for identical output quality. That gap is why the discipline exists as a specialty at all.

The physics driving this is worth stating plainly, because most of the operational differences downstream are consequences of it. A transformer generating text does two jobs. The first — prefill — processes the entire input prompt in parallel. It is compute-bound: the GPU's matrix units run hot, arithmetic intensity is high, and throughput scales close to the accelerator's FLOPs rating. The second — decode — generates output tokens one at a time, each step requiring a full read of the model weights and the accumulated key-value cache from high-bandwidth memory. Decode is memory-bandwidth-bound. A single decode step for a 70B-parameter model in bf16 must stream roughly 140 GB of weights per forward pass; on an accelerator with ~3 TB/s of HBM bandwidth, that sets a hard floor of about 45 ms per step regardless of how much compute is idle. Batching amortizes that weight read across many sequences, which is why decode throughput is fundamentally a batching problem and prefill throughput is fundamentally a compute problem.

Teams that internalize this stop treating "the GPU" as a single fungible resource. They start asking which phase is starving, and they provision for the two answers separately.

What makes 2027 different from 2025 is less any single breakthrough than the accumulation of workload shifts. Reasoning models that emit long internal chains before answering pushed the output-token-to-input-token ratio up by an order of magnitude for a meaningful slice of traffic, which moved the cost center from prefill to decode and made KV cache memory the scarcest resource in the rack. Agentic workloads made request patterns bursty and multi-turn, with each turn re-sending a conversation history that is 90%+ identical to the prior turn. Long-context windows made prefill costs quadratic-ish in ways that punished naive re-processing. And the fleet itself got heterogeneous — a mix of accelerator generations, some inherited, some rented — so schedulers had to become hardware-aware.

The adjacent effect that surprises finance teams: inference cost has stopped tracking headcount or user count and started tracking *product design*. A feature that adds one more agent hop, or one more retrieval-augmented context block, changes the unit economics more than a 20% traffic increase does. Infrastructure teams that report cost per successful task give product managers a steering wheel. Teams that report GPU hours give them a fog bank.

The step-by-step process

The operating loop most mature teams converge on looks less like a deployment pipeline and more like a continuous control system. Here is the sequence, with the concrete decisions at each stage.

Characterize the workload before choosing hardware. Pull two weeks of production traces and compute the distribution — not the mean — of input tokens, output tokens, and inter-turn arrival gaps. The input:output ratio is the single most decision-relevant number you will produce. A retrieval-heavy summarization workload at 8000:200 is prefill-dominated and wants raw FLOPs. A reasoning-agent workload at 1500:4000 is decode-dominated and wants memory bandwidth and KV capacity. Most teams discover they have three or four distinct workload classes wearing one trench coat, and the first real win is simply routing them to different pools.

Set service objectives on the right two metrics. Time-to-first-token (TTFT) is a prefill metric. Time-per-output-token (TPOT), sometimes called inter-token latency, is a decode metric. They trade against each other and they are tuned by different knobs, so a single "p95 latency" SLO is nearly useless. A conversational UI might target TTFT under 400 ms and TPOT under 40 ms — the latter being roughly the threshold above which streamed text starts reading slower than a person does. A batch enrichment job might accept 30-second TTFT and care only about tokens per dollar. Writing these down separately is what unlocks running them on different infrastructure.

Pick the parallelism layout to fit memory, then tune for latency. Tensor parallelism splits each layer across GPUs inside one node; it cuts per-step latency but adds an all-reduce on every layer, so it degrades badly across slow interconnects. Pipeline parallelism splits layers across nodes; it tolerates slower links but adds bubbles unless you have enough in-flight microbatches. Expert parallelism, for mixture-of-experts models, shards experts across devices and introduces an all-to-all whose cost depends on routing entropy. The practical rule: use tensor parallelism up to the boundary of a high-bandwidth interconnect domain, then pipeline beyond it, and size the degree so that model weights plus peak KV cache fit with 15–25% headroom. Running at 95% memory occupancy means the scheduler starts preempting under any traffic spike.

Turn on continuous batching and give the scheduler room to work. Static batching — wait for N requests, run them lockstep, return together — wastes enormous capacity because sequences finish at different times and the whole batch runs until the longest one completes. Continuous (in-flight) batching admits new requests at every decode step as slots free up. This is table stakes in every serious serving stack now, but the second-order tuning is where teams differ: max batch size, max number of scheduled tokens per iteration, and the prefill/decode interleaving policy. Chunked prefill — splitting a long prompt into pieces and interleaving them with decode steps — keeps a long prompt from stalling every in-flight generation, at the cost of slightly worse TTFT for that one request.

Make the KV cache a managed resource, not an accident. Paged attention allocates KV cache in fixed-size blocks rather than contiguous per-sequence buffers, cutting fragmentation waste from the 60–80% range down to low single digits. On top of that, prefix caching reuses the computed KV blocks for any shared prompt prefix — system prompt, tool definitions, conversation history, retrieved document. In multi-turn agent traffic, prefix hit rates of 70–90% are ordinary, and each hit removes essentially the entire prefill cost of that turn. This is the highest-leverage single optimization available to most teams, and it is mostly a matter of routing: hash the prefix and send the request to the replica that already holds those blocks, rather than round-robining it to a cold one.

Disaggregate prefill from decode once volume justifies it. Running both phases on the same GPU means a long prefill periodically freezes decoding for every co-resident sequence, which shows up as ugly TPOT jitter. Splitting them into separate pools — prefill workers that produce KV cache and stream it over the interconnect to decode workers — lets each pool be sized, scaled, and even hardware-matched independently. Prefill pools want compute-dense parts; decode pools want bandwidth and memory capacity, which sometimes means older or cheaper accelerators do the job fine. The cost is a KV transfer on every request, so this pays off above a volume threshold and hurts below it.

Quantize deliberately and measure quality, not just perplexity. Weight-only 8-bit quantization is close to free in quality terms and roughly halves the memory-bandwidth bill for decode. FP8 for both weights and activations is well supported on current accelerators and typically lands within noise on most benchmarks. 4-bit gets aggressive — often fine for chat, often not fine for code generation, structured extraction, or long-chain reasoning where small errors compound. KV cache quantization to 8 bits is separately valuable because it directly increases how many concurrent sequences fit. The discipline that separates teams: run your *own* task evals before and after, on real production prompts, and treat a 1–2% drop on the eval that matters as a blocker even if public benchmarks show nothing.

Route across a model portfolio. Almost no mature deployment serves one model. A cascade sends everything to a small fast model first, checks a confidence or verifier signal, and escalates only the fraction that needs the large model. Typical escalation rates land somewhere in the 10–30% band for well-characterized tasks, which is where the headline cost reductions come from. Speculative decoding is the same idea inside a single response: a small draft model proposes several tokens, the large model verifies them in one batched pass, and acceptance rates of 60–80% on predictable text yield 1.5–3x decode speedups with mathematically identical output.

Close the loop with autoscaling on queue depth. GPU utilization is a misleading scaling signal — a decode-bound worker can sit at 90% "utilization" while doing almost nothing useful, because the metric counts kernel occupancy, not work. Scale on time-in-queue or on the ratio of pending tokens to serving capacity. Budget for cold starts honestly: loading a large model into GPU memory takes tens of seconds to minutes depending on weight size and storage path, so keep warm capacity for the burst you actually get rather than the average you report.

Costs, timelines, and typical ranges

Numbers here are ranges and ratios rather than prices, because accelerator pricing moves quarterly and any specific figure ages badly. The ratios are what survive.

The dominant cost term is decode, and it is set by memory bandwidth. For a given accelerator, decode throughput in tokens per second is approximately (memory bandwidth ÷ bytes-per-forward-pass) × batch size, until you run out of KV cache room or hit compute limits. Two consequences follow. First, halving model precision roughly doubles decode throughput — this is why quantization is the first lever, not the last. Second, doubling batch size is nearly free in tokens-per-second-per-dollar terms right up until KV memory runs out, which is why KV cache capacity is the real ceiling on serving economics.

Batch size effects are large and nonlinear. Moving from batch size 1 to 32 typically improves tokens per dollar by an order of magnitude, because you amortize the same weight read across 32 sequences. Beyond that, gains flatten and per-request latency starts climbing. The practical sweet spot for interactive workloads is usually where TPOT sits just inside your SLO — push batch size up until inter-token latency starts violating the target, then back off one step. For offline batch jobs with no latency SLO, push until memory is the constraint.

Prefix caching pays back fastest. In agentic and multi-turn traffic, where each turn resends a long shared history, hit rates of 70–90% are typical, and each hit eliminates most of that turn's prefill work. For workloads with long system prompts and tool schemas — common in agent deployments where the schema block alone can run several thousand tokens — this alone often cuts total compute 30–50%. Implementation effort is days, not quarters, if your serving stack supports it and you can make the router prefix-aware.

Quantization gives roughly linear memory-bandwidth returns. bf16 to FP8 halves weight traffic. FP8 to 4-bit halves it again, with steeper quality risk. Because decode is bandwidth-bound, these translate close to proportionally into throughput. Expect a week or two of work, most of it evaluation rather than implementation.

Routing and cascades produce the biggest headline numbers and carry the most risk. If 70% of your traffic can be answered by a model that costs a fraction of the frontier one, the arithmetic is obvious. The risk is that the classifier is wrong on the tail, and the tail is where user trust lives. Budget several weeks to build the evaluation harness that tells you the escalation threshold is safe — the harness is the real deliverable, not the router.

Disaggregation is a bigger lift. Weeks to months, meaningful operational complexity, and it only pays above a traffic volume where the jitter from co-located prefill is actually costing you. Below that, it is complexity you will regret.

Utilization targets are worth stating. Well-run fleets keep GPUs busy 60–80% of the time on real work, with the remainder as headroom for burst. Chasing 95% means your p99 latency becomes a function of someone else's traffic spike. Anything under 30% sustained is a routing or bin-packing problem, not a capacity problem, and buying more hardware will not fix it.

Cost per successful task is the metric that changes behavior. Cost per token is easy to compute and nearly useless for decisions, because a cheap model that fails and triggers a retry plus an escalation costs more than the expensive model would have. Instrument the outcome — did the extraction validate, did the agent complete, did the user accept the suggestion — and divide total spend by successes. Teams that make this switch commonly find that their cheapest-per-token path is not their cheapest-per-outcome path.

Where teams get it wrong

Optimizing the wrong phase. A team stares at low GPU compute utilization during decode and concludes they need a bigger batch or faster kernels. But decode is bandwidth-bound; the compute units are *supposed* to be idle. Chasing FLOPs utilization there produces months of work and no improvement. Diagnose the bound before optimizing: if throughput scales linearly with batch size, you are bandwidth-bound and should quantize or batch harder; if it doesn't, you are compute-bound or scheduler-bound.

Treating GPU utilization as the health metric. It conflates two very different states and it is the most common cause of both over- and under-provisioning. Queue wait time and token throughput per dollar are the metrics that correlate with what you actually care about.

Load balancing round-robin across a prefix-cached fleet. This is a quiet, expensive mistake. A stateless load balancer will happily send turn 5 of a conversation to a replica that has never seen turns 1–4, throwing away a cache hit that would have eliminated most of the prefill. The fix is a prefix-aware or session-affine router, and the effect on a multi-turn workload is often larger than any kernel-level optimization.

Benchmarking with uniform synthetic traffic. Load tests that send 1000-token prompts and expect 100-token completions, at a steady rate, tell you almost nothing about a fleet that will face a heavy-tailed real distribution with bursts. The tail is what breaks: one 100K-token request can occupy KV cache that would have served dozens of normal ones. Replay real traces, or at minimum sample from your real length distribution and your real arrival pattern.

Quantizing on public benchmark evidence. "FP8 shows no degradation" is true in aggregate and can be false for your task. Structured output, code, and long-chain reasoning degrade differently than open-ended chat. Always re-run your own evals.

Ignoring the tokenizer and the prompt. A prompt with 30% redundant boilerplate costs 30% more prefill forever. Deduplicating retrieved context, trimming few-shot examples that no longer earn their place, and compacting tool schemas are unglamorous and frequently produce double-digit savings with zero infrastructure change. This is the upstream lever: the cheapest token is the one you never send.

No admission control. When demand exceeds capacity, a system with no queue limit degrades for everyone simultaneously — every request gets slower, and clients time out and retry, which adds load. A system with admission control sheds or queues the lowest-priority traffic and keeps the rest inside SLO. Decide in advance which traffic classes are droppable.

Forgetting that retries are traffic. Client-side retry logic on a saturated fleet is an amplifier. Exponential backoff with jitter, and a circuit breaker, belong in the client library, not in the postmortem.

Under-instrumenting the KV cache. If you cannot see cache hit rate, eviction rate, and preemption count on a dashboard, you are flying blind on the resource that most determines your economics. Preemption in particular is insidious — a preempted sequence has to recompute its prefill, so a fleet running near memory capacity silently burns compute redoing work.

Decision framework: when to choose what

The ordering below reflects effort-to-payoff, and it is deliberately conservative: do the cheap high-leverage things before the architectural ones.

Start with prompt and context hygiene. Zero infrastructure risk, immediate effect, and it improves every downstream optimization's baseline. Trim, dedupe, compact.

Then prefix caching plus a cache-aware router. Highest leverage per unit of effort for any multi-turn or shared-system-prompt workload. If your traffic is single-shot and diverse, skip it — the hit rate won't justify the routing complexity.

Then quantization, gated on your own evals. FP8 first. Only go to 4-bit if evals hold and memory is genuinely your ceiling.

Then continuous batching tuning and chunked prefill. Assuming your stack already does continuous batching, the tuning is scheduler parameters. Chunked prefill matters specifically when you have a wide spread of input lengths and long prompts are visibly stalling short ones.

Then model routing and cascades. Big savings, but the prerequisite is an evaluation harness good enough to trust the escalation threshold. Build the harness first; the router is the easy part.

Then speculative decoding. Best when output text is predictable — structured formats, code with lots of boilerplate, templated responses. Weak on high-entropy creative generation where the draft model's acceptance rate collapses and you pay for verification with no speedup.

Then prefill/decode disaggregation. Only above real volume, and only when co-located prefill is demonstrably causing TPOT jitter you can measure. Otherwise it is operational complexity without a return.

Then hardware heterogeneity. Once you know which pool is bandwidth-bound and which is compute-bound, you can buy or rent differently for each, and route older accelerators to the workloads that don't need the newest part. This is where inherited fleets stop being a liability.

Two adjacent decisions worth naming, because they sit just outside the inference boundary but dominate the same budget. First, build versus buy: self-hosting only makes economic sense above a sustained volume where you can keep the fleet busy — a pool at 20% utilization loses to an API on price, every time, and adds an on-call rotation. The crossover is a function of your traffic floor, not your traffic peak. Second, fine-tuning as an inference optimization: a smaller fine-tuned model that matches a larger general model on your narrow task changes the serving economics permanently, and the cost is a training run plus a maintenance obligation. Teams frequently reach for serving tricks when the real answer was a smaller specialized model all along.

How this changes the org, not just the stack

The organizational shift is the part that gets underestimated. Inference optimization sits at a seam between three groups that historically did not share a dashboard: the ML team that picks models, the platform team that runs hardware, and the product team that decides how many model calls a feature makes. When those groups optimize independently, the ML team ships a better model that costs 3x, the platform team buys capacity to absorb it, and the product team adds an agent hop that doubles it again — and nobody made a bad decision in isolation.

The teams that handle this well do a few concrete things differently. They publish a shared cost-per-successful-task dashboard that all three groups read. They make inference cost a line item in feature design review, so a product manager sees the marginal cost of the second retrieval pass before committing to it. They keep an eval harness that both ML and platform trust, because every serving optimization — quantization, routing, speculative decoding — is a quality bet, and without a shared source of truth the argument is unresolvable and the safe default is to skip the optimization.

The downstream effect on adjacent workflows is real too. Data teams find that retrieval quality is now an infrastructure lever: better chunking means fewer retrieved documents, which means shorter prefill, which means lower cost and lower TTFT simultaneously. Security teams find that prefix caching creates a shared-state surface that needs tenant isolation. Finance teams find that a reserved-capacity commitment behaves like a fixed cost against a workload whose variance they don't yet understand, which is an argument for keeping a burst path to an API even when self-hosting is the primary route.

None of that is exotic. It's the ordinary consequence of a cost center that used to be small becoming large enough that its shape shows up in the P&L.

Related questions

Why is decode slower than prefill per token?

Prefill processes all input tokens in parallel and saturates compute units. Decode generates one token at a time, and each step must re-read the entire model weights plus KV cache from memory. That makes decode bound by memory bandwidth, not compute, so per-token cost stays high regardless of idle FLOPs.

Does continuous batching hurt individual request latency?

Slightly, and it is almost always worth it. Adding sequences to a batch increases per-step time modestly while multiplying total throughput. The risk is over-batching past your inter-token latency target. Tune batch size upward until TPOT approaches the SLO ceiling, then step back one increment.

When is speculative decoding not worth it?

When output entropy is high. Draft-model acceptance rates collapse on creative or unpredictable text, and you pay verification compute for few accepted tokens. It shines on structured output, code with boilerplate, and templated responses where acceptance commonly runs 60–80%.

Should we self-host or use an inference API?

Self-hosting wins above a sustained traffic floor high enough to keep the fleet busy — a pool idling at 20% utilization loses to an API on price and adds on-call burden. Many teams run both: self-hosted for the steady base, API for burst and for frontier-model escalation.

What single metric best predicts inference spend?

Total output tokens, weighted by model tier. Output tokens drive decode, decode dominates cost, and model tier sets the per-token rate. Track it alongside cost per successful task so that quality-driven retries and escalations show up rather than hiding inside a flattering per-token average.

FAQ

What does "disaggregated serving" actually mean in practice?

It means running prefill and decode on separate worker pools instead of the same GPU. A prefill worker processes the prompt, produces the KV cache, and ships it over the interconnect to a decode worker that generates the output. Each pool scales independently and can sit on different hardware — compute-dense parts for prefill, bandwidth-and-memory-heavy parts for decode. The benefit is that a long prompt no longer freezes decoding for everyone else on that GPU, which shows up as much steadier inter-token latency. The cost is the KV transfer on every request plus real operational complexity, so it earns its place only above meaningful volume.

How much does prefix caching typically save?

It depends entirely on how much of your traffic shares a prefix. Multi-turn conversations, agent loops that resend tool schemas, and applications with long fixed system prompts routinely see hit rates in the 70–90% range, and each hit removes essentially all of that turn's prefill work. Single-shot workloads with diverse prompts see close to nothing. The prerequisite most teams miss is routing: a round-robin load balancer will send a request to a replica with a cold cache and silently discard the benefit.

Is 4-bit quantization safe for production?

Sometimes. It halves memory traffic again relative to 8-bit, which matters when decode bandwidth is your ceiling. But degradation is uneven — conversational quality often holds up while code generation, structured extraction, and long multi-step reasoning degrade measurably, because small errors compound across a long output. The only reliable answer comes from running your own task evals on real production prompts, before and after. Treat public benchmark parity as a hypothesis, not evidence.

Why is GPU utilization a bad autoscaling signal?

Because it measures kernel occupancy, not useful work. A decode-bound worker can report high utilization while being starved on memory bandwidth and producing few tokens, and a lightly loaded worker can report low utilization while comfortably inside SLO. Scale instead on queue wait time or on pending tokens relative to serving capacity — signals that move when user experience moves. Keep utilization on the dashboard as a diagnostic, just not as the trigger.

What is chunked prefill and when do I need it?

It splits a long prompt's prefill into smaller pieces and interleaves them with decode steps rather than running the whole prefill as one blocking operation. Without it, one very long prompt stalls token generation for every other in-flight request on that GPU, producing visible stutter. You need it when your input length distribution has a long tail — a mix of short chat turns and occasional huge document loads. The trade-off is slightly worse time-to-first-token for the chunked request itself.

How do I know whether I am compute-bound or bandwidth-bound?

Run a batch-size sweep. If throughput climbs close to linearly as you raise batch size, you are bandwidth-bound — you are amortizing the same weight read across more sequences, so quantization and larger batches are your levers. If throughput flattens early, you are compute-bound or scheduler-bound, and the answer lies in kernel efficiency, parallelism layout, or scheduling policy instead. Do this before committing to any optimization program; it saves months.

Sources

• https://arxiv.org/abs/2309.06180 — Efficient Memory Management for Large Language Model Serving with PagedAttention
• https://arxiv.org/abs/2211.05102 — Efficiently Scaling Transformer Inference
• https://arxiv.org/abs/2302.01318 — Accelerating Large Language Model Decoding with Speculative Sampling
• https://arxiv.org/abs/2211.17192 — Fast Inference from Transformers via Speculative Decoding
• https://arxiv.org/abs/2401.09670 — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving
• https://docs.vllm.ai/en/latest/ — vLLM documentation
• https://github.com/NVIDIA/TensorRT-LLM — NVIDIA TensorRT-LLM
• https://pytorch.org/blog/accelerating-generative-ai-2/ — Accelerating Generative AI with PyTorch II: GPT, Fast
• https://huggingface.co/docs/text-generation-inference/index — Hugging Face Text Generation Inference

Related on PULSE

• How do you calculate the true cost per API call in an AI-powered product?
• What changes when a sales team adds an AI agent to the pipeline?
• How do RevOps teams budget for variable AI usage costs?
• What should a build-versus-buy analysis include for AI features?
• How do you measure whether an AI feature actually improved outcomes?