How do you choose between CPU and GPU for model inference in 2027?

What it is and why it matters

The CPU-versus-GPU question sounds like a hardware procurement detail. It is actually a unit-economics decision that determines whether an AI feature inside your product ships at a gross margin you can defend. Every inference call has a cost, and that cost lands on the same P&L line as your hosting bill. When a revenue team ships an AI-powered lead scorer, a call summarizer, or an inbox triage agent, the compute substrate underneath it decides whether that feature costs $200 a month or $20,000 a month at scale.

The technical distinction is simpler than the marketing around it. A CPU has a small number of very fast, very general cores — typically 8 to 128 in a server-class part — each capable of complex branching and out-of-order execution. A GPU has thousands of simple cores optimized for the same arithmetic operation applied across large blocks of data simultaneously. Transformer inference is dominated by matrix multiplication, which is exactly the shape of work GPUs were built for. That is the whole story in one sentence, and everything else is qualification.

The qualification matters, though, because transformer inference has two distinct phases with opposite hardware profiles. The prefill phase — processing the input prompt — is compute-bound. It multiplies large matrices and saturates arithmetic units, and a GPU crushes a CPU here, often by one to two orders of magnitude. The decode phase — generating output tokens one at a time — is memory-bandwidth-bound. Each new token requires streaming the entire set of model weights from memory through the arithmetic units, and the bottleneck is how fast memory can feed the compute, not how much compute exists.

That second fact is why the naive "GPUs are 100x faster" claim collapses in practice. For single-stream decode of a small model, a modern server CPU with high-bandwidth DDR5 and AVX-512 or AMX instructions can be within a small multiple of an entry-level GPU, not a hundred times slower. Server memory bandwidth sits in the low hundreds of GB/s across many channels; high-end accelerator memory sits in the multiple-TB/s range. The ratio between those two numbers, not the FLOPS ratio, predicts single-stream decode speed for a memory-bound workload.

Where GPUs pull decisively ahead is concurrency. Continuous batching — the technique used by every serious modern serving stack — lets a GPU process many concurrent requests through the same weight-streaming pass. If you read the weights once and use them to advance sixty-four sequences instead of one, your effective per-request cost drops by close to that factor until you saturate compute or memory capacity. The GPU's advantage is therefore not primarily speed; it is amortization. A CPU can batch too, but its far smaller memory bandwidth and cache hierarchy mean the amortization curve flattens much sooner.

Quantization changes the arithmetic on both sides. Running a model at 8-bit or 4-bit weights roughly halves or quarters the bytes that must move per token, which directly multiplies decode throughput on a bandwidth-bound device. It also shrinks the memory footprint enough that models which would not fit in a small GPU's memory suddenly do, and models that would have thrashed a CPU's cache suddenly stream cleanly. Any honest CPU-versus-GPU comparison must specify precision, because a 4-bit CPU deployment and a 16-bit GPU deployment are not comparing the same workload.

The adjacent question that often matters more is whether you should be self-hosting at all. For a large share of revenue-operations use cases — summarizing a call, extracting fields from an email, classifying an inbound lead — a hosted API endpoint is cheaper than any self-managed deployment until volume is genuinely high and sustained. Self-hosting buys you data residency, latency floors, predictable unit cost at scale, and freedom from rate limits. It costs you an on-call rotation, capacity planning, and model-update work. Choose the CPU-or-GPU question only after you have honestly answered the build-or-buy one.

The step-by-step process

Start by writing down the workload before you look at any hardware. You need five numbers: model size in parameters and the precision you intend to serve at; median and p95 input token count; median and p95 output token count; requests per second at peak, not average; and the latency target expressed as time-to-first-token and tokens-per-second-after-first. Teams that skip this step end up buying accelerators to serve four requests an hour.

Second, compute the memory floor. Weights consume roughly two bytes per parameter at 16-bit, one byte at 8-bit, and around half a byte at 4-bit, plus overhead. On top of weights you need KV cache, which scales with layers, heads, head dimension, batch size, and sequence length — and which is frequently the thing that actually runs you out of memory in production, not the weights. A long-context workload with many concurrent sessions can spend more memory on KV cache than on the model itself. If your total does not fit in a candidate device's memory, that device is out unless you are willing to shard across multiple accelerators or offload layers, both of which cost latency.

Third, classify the workload as latency-sensitive or throughput-sensitive. Interactive workloads — a chat surface, an in-CRM assistant, a live call copilot — are latency-sensitive; a human is waiting and time-to-first-token under a second or so is table stakes. Batch workloads — nightly enrichment of a lead database, re-scoring an entire pipeline, backfilling summaries across a year of call recordings — are throughput-sensitive; nobody is watching and the only thing that matters is total cost to completion. These two classes frequently justify different hardware inside the same company.

Fourth, benchmark on your own traffic shape rather than trusting published numbers. Replay a representative sample of real prompts, not synthetic fixed-length ones, because prefill cost scales with actual input length and your real inputs are probably longer and more variable than a benchmark's. Measure p50, p95, and p99 latency under concurrency that matches your peak, and record throughput in tokens per second alongside utilization. Run both candidates with the same quantization and the same serving stack where possible so the comparison is honest.

Fifth, convert every measurement into cost per thousand tokens. Take the hourly cost of the instance, divide by the tokens per second it sustained at your target latency, and normalize. This single number makes CPU and GPU directly comparable and is the only figure that translates cleanly into a margin conversation with finance. It also exposes the trap of an underutilized accelerator: a fast device running at fifteen percent utilization can easily lose to a slower device running near saturation.

Sixth, decide the deployment topology, which is a separate question from the substrate. A GPU that sits idle twenty-two hours a day is a terrible purchase even if it wins every latency benchmark, which is why serverless or per-second-billed accelerator capacity often beats a reserved instance for spiky traffic. Conversely, sustained high-volume traffic inverts that: reserved or owned hardware amortizes far better than per-request pricing once utilization is consistently high. The break-even is a utilization threshold, and you should compute it rather than guess it.

Seventh, plan the fallback path before you need it. Accelerator capacity is not always available in the region you want, at the price you want, on the day you want it. A serving architecture that can degrade to a smaller CPU-served model — or to a hosted API — under capacity pressure is worth building early, because retrofitting it during an outage is miserable. Route by request class: interactive traffic gets the fast path, batch traffic gets whatever is cheapest and can wait.

Costs, timelines, and typical ranges

Talk about cost in three buckets: compute, engineering, and opportunity. Compute is the visible one. CPU instances on major clouds are priced roughly an order of magnitude below GPU instances of comparable vintage, and small accelerators sit well below flagship data-center parts. Exact prices move constantly and vary by region, commitment, and generation, so treat any specific figure you read as a starting point for your own quote rather than a fact. What is stable is the shape: per-hour accelerator cost is high, per-hour CPU cost is low, and the winner depends entirely on how many tokens each produces per hour at your latency bar.

The utilization math is where most of the decision lives. Consider a workload that must serve some sustained token volume. If a GPU instance costs roughly ten times a CPU instance per hour but delivers twenty times the throughput at your required latency, the GPU is half the cost per token and you should take it. Flip one variable — say your traffic only keeps the GPU busy a quarter of the time because it arrives in short bursts — and the effective throughput advantage collapses to five times against ten times the price, and the CPU wins. This is why utilization, not benchmark speed, is the number to argue about.

Engineering cost is routinely underestimated. A self-hosted GPU serving stack means driver and runtime version management, container images that are large and finicky, model loading times measured in tens of seconds that complicate autoscaling, and a real on-call burden when a node goes unhealthy. CPU serving is meaningfully simpler operationally — it runs on the same commodity instances as the rest of your services, scales with the autoscaler you already have, and does not require a specialist. Budget several engineer-weeks for a first production GPU deployment and ongoing maintenance thereafter; budget a fraction of that for CPU.

Timelines follow the same asymmetry. A CPU-served small model behind an existing service can often go from decision to production in days, because there is no new infrastructure primitive involved. A GPU deployment typically runs weeks: capacity request or quota increase, image build, serving-stack selection and tuning, load testing, autoscaling policy, observability, and a rollback plan. If your goal is to validate whether an AI feature moves a revenue metric at all, the fast path is almost always a hosted API or a small CPU-served model, with the GPU decision deferred until the feature has proven it deserves the investment.

Model size gives useful rules of thumb, stated as ranges rather than hard lines. Models in the sub-1B range — classifiers, embedding models, small extractors, rerankers — run comfortably on CPU at practical latencies and are frequently the wrong thing to put on an accelerator at all. The 1B to 7B range is the genuine gray zone: CPU works for low concurrency and tolerant latency, especially at 4-bit, while GPU wins as soon as concurrency rises. Above roughly 7B to 13B, CPU serving becomes impractical for anything interactive, and above that, memory capacity rather than speed is usually the binding constraint.

Embedding and retrieval workloads deserve their own note because they behave differently. Embedding generation is pure prefill with no decode phase, which makes it highly parallel and unusually GPU-friendly per unit of work — but embedding models are small, so a CPU fleet often delivers adequate throughput at lower cost, particularly for incremental updates rather than full-corpus rebuilds. A one-time backfill of millions of documents is a strong GPU case; keeping that index warm with a few thousand daily updates usually is not. The same split applies to reranking, where models are small and latency budgets are tight but volumes are modest.

Finally, count the cost of being wrong in each direction. Over-provisioning to GPU wastes money but the feature works; under-provisioning to CPU produces slow responses, which users experience as a broken product and which quietly suppress adoption of the feature you just built. If the workload is genuinely borderline and the feature is customer-facing, the asymmetry favors the faster substrate — you can always migrate down once you have real traffic data, and that migration is far less painful than winning back users who decided the feature was sluggish.

Where teams get it wrong

The most common error is benchmarking the wrong phase. A team measures prompt processing speed, sees a dramatic GPU win, and buys accelerators for a workload that is ninety percent short-prompt decode where the gap is far narrower. The inverse also happens: a team measures single-stream decode, concludes CPU is nearly as good, and deploys it into a service that immediately hits fifty concurrent requests and falls over. Always benchmark the phase mix and the concurrency level you will actually serve.

The second error is ignoring KV cache growth. Memory planning done on weights alone looks fine in staging with short prompts and one user, then collapses in production when long documents and dozens of concurrent sessions arrive. KV cache scales with both batch size and sequence length, and long-context features are exactly the ones product teams love to add after launch. Size for your p95 context length at your peak concurrency, not your median at your average.

Third: comparing across different precisions and calling it a hardware result. A 4-bit CPU deployment beating a 16-bit GPU deployment on cost per token is a quantization result wearing a hardware costume. Fix precision across both arms of the test, then vary it deliberately as its own experiment. Also verify output quality at each precision on your actual task — aggressive quantization can degrade structured extraction and instruction-following noticeably even when perplexity barely moves, and a cheaper wrong answer is not a saving.

Fourth: treating average utilization as the planning number. Traffic in revenue tooling is famously spiky — Monday-morning pipeline reviews, end-of-quarter crunches, the hour after a marketing send. Provisioning for the average and hoping means your p99 lands during exactly the moments that matter most to the business. Provision against the peak you actually observe, then attack the cost of that peak with autoscaling, request queuing, or by routing non-urgent work to a batch lane.

Fifth: forgetting that model choice is a bigger lever than hardware choice. Teams will spend a month optimizing serving infrastructure for a 13B model when a well-prompted 3B model, or a small fine-tune, would have hit the quality bar at a fraction of the cost on hardware they already own. Always run the "can a smaller model do this" experiment before the "which chip" experiment, because it frequently makes the hardware question disappear entirely. Distillation and task-specific fine-tuning are underused here — a small model trained on your task often beats a large general model at it.

Sixth: no cost observability. If you cannot answer "what did inference cost us last week, broken down by feature," you cannot manage the decision at all. Instrument token counts per request, tag them by feature and customer tier, and put cost per thousand tokens on a dashboard next to latency. This is the same discipline revenue teams already apply to CAC and payback — inference is now a cost of goods sold, and it should be governed like one. Without it, the first real bill becomes a fire drill.

Seventh: single-substrate thinking. The question is framed as CPU *or* GPU, but mature deployments are usually both. Small classifiers, guardrail checks, embedding lookups, and routing decisions live on CPU alongside the application; the heavy generation path lives on accelerators; overnight batch jobs run on whatever is cheapest. Designing the serving layer so request class determines route is more work upfront and dramatically cheaper at steady state.

Decision framework: when to choose what

Reduce the decision to a short ordered sequence of gates, and stop at the first one that answers you. Gate one: does the model fit in CPU-accessible memory at your chosen precision, with KV cache headroom for peak concurrency? If not, the decision is made — accelerator, or a smaller model. Gate two: is anyone waiting on the response? If nobody is, the workload is batch and you should pick purely on cost per token at whatever latency completes the job inside its window. Gate three: what is peak concurrency? Below a handful of simultaneous requests, CPU stays competitive for small models; well above that, batching economics favor the GPU decisively.

Apply the framework to concrete revenue-operations shapes and it resolves quickly. Real-time call transcription and live coaching: interactive, concurrent, latency-critical — accelerator or hosted API, no debate. Nightly enrichment of a hundred thousand CRM records: batch, cost-driven, and a CPU fleet or spot accelerator capacity both work; pick on price. Inbound lead classification with a small fine-tuned model at a few requests per second: CPU on the instances you already run, and stop thinking about it. Embedding refresh for a knowledge base: GPU for the initial backfill, CPU for the daily delta.

The same logic extends to neighboring domains that revenue teams touch. Document extraction pipelines in finance and legal ops look like batch inference with long inputs and heavy prefill, which shifts them toward accelerators more than their modest volume would suggest. Support-ticket triage looks like high-volume short-prompt classification, which is a CPU sweet spot. Forecasting and scoring models that are not transformers at all — gradient-boosted trees, classical regressions — belong on CPU essentially without exception, and it is worth checking whether the problem in front of you actually needs a language model before the hardware question is even on the table.

Revisit the decision on a schedule rather than treating it as permanent. Hardware generations, quantization techniques, serving-stack efficiency, and hosted-API pricing all move faster than most infrastructure decisions, and a choice that was correct eighteen months ago may be expensive today. A quarterly review that re-runs the cost-per-thousand-tokens benchmark against current options takes a day and routinely finds double-digit percentage savings. Tie that review to the same cadence you use for other vendor and infrastructure spend so it actually happens.

Build the escape hatches in from the start. Keep the model interface behind an abstraction so the substrate can change without touching product code. Keep prompts, model versions, and routing rules in configuration rather than compiled into services. Keep a benchmark harness that replays real traffic and can be pointed at any candidate in an afternoon. These three habits turn what feels like an irreversible architecture commitment into a reversible operational choice, which is the actual goal — you want to be able to choose again cheaply when conditions change.

Related questions

Does quantization change the CPU-versus-GPU answer?

Substantially. Moving from 16-bit to 4-bit weights cuts the bytes streamed per token by roughly four times, which directly multiplies decode throughput on bandwidth-bound hardware and shrinks the memory floor. It can move a model from "impossible on CPU" to "acceptable," so always fix precision before comparing substrates.

When is a hosted API cheaper than self-hosting either one?

Almost always at low or spiky volume. Hosted endpoints charge per token with no idle cost, so they win until your utilization is high and sustained enough that a dedicated instance amortizes better. Self-host when you need data residency, a hard latency floor, or predictable unit cost at real scale.

What about NPUs and other accelerators?

They occupy a middle ground — better throughput-per-watt than CPU for small models, less flexible and less software-mature than GPUs. Evaluate them the same way: benchmark your real traffic, compute cost per thousand tokens, and check that your serving stack and quantization format are genuinely supported before committing.

How much does context length affect the choice?

A great deal. Long inputs make prefill dominant, which favors accelerators, and they inflate KV cache, which consumes memory that would otherwise fund concurrency. A workload with 500-token prompts and one with 32,000-token prompts are different hardware problems even at identical request rates.

Should batch and interactive traffic share hardware?

Usually not. Batch work will fill queues and inflate p95 for interactive users. Separate the lanes — route interactive traffic to a latency-tuned pool and batch work to a cost-tuned pool, possibly on entirely different substrates — and let each be sized against its own objective.

FAQ

Is a GPU always faster than a CPU for inference?

No. For compute-bound prefill on large models, GPUs win overwhelmingly. For single-stream decode of a small quantized model, a modern server CPU with wide vector or matrix instructions and high memory bandwidth can land within a small multiple of an entry-level accelerator. The GPU's durable advantage shows up under concurrency, where continuous batching amortizes weight streaming across many simultaneous requests.

What model size is the practical CPU ceiling?

There is no hard line, but sub-1B models are comfortable on CPU for interactive use, 1B to 7B is a genuine gray zone that depends on quantization and concurrency, and above roughly 7B to 13B, CPU serving stops being practical for anything a human is waiting on. Batch workloads can push higher because latency does not bind.

How do I compare CPU and GPU options fairly?

Fix the model, the precision, and the serving stack across both arms. Replay real production traffic rather than synthetic fixed-length prompts. Measure p50, p95, and p99 latency at your actual peak concurrency, then convert sustained throughput and instance price into cost per thousand tokens. That single normalized number is what makes the comparison meaningful and what finance will understand.

Why does utilization matter more than raw speed?

Because you pay for the instance whether or not it is busy. A device that is three times faster but idle three-quarters of the time delivers no cost advantage at all. Compute effective cost per token using observed utilization on your real traffic pattern, and use serverless or per-second billing when traffic is too spiky to keep dedicated capacity busy.

Can one deployment use both CPU and GPU?

Yes, and mature systems usually do. Route by request class: small classifiers, guardrails, routing logic, and embedding lookups on CPU; heavy generation on accelerators; overnight batch jobs on whatever capacity is cheapest at the time. Keep the model interface behind an abstraction so the routing rules live in configuration and the substrate can change without product code changes.

How often should we revisit this decision?

Quarterly is a reasonable default. Hardware generations, quantization methods, serving-stack efficiency, and hosted-API pricing all move quickly, so a choice that was optimal a year ago is often expensive now. Keep a benchmark harness that replays real traffic against any candidate in an afternoon, and fold the review into your normal infrastructure spend cycle.

Sources

• https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
• https://docs.vllm.ai/en/latest/
• https://huggingface.co/docs/transformers/llm_tutorial_optimization
• https://pytorch.org/docs/stable/quantization.html
• https://github.com/ggml-org/llama.cpp
• https://docs.nvidia.com/deeplearning/tensorrt-llm/latest/index.html
• https://onnxruntime.ai/docs/performance/
• https://aws.amazon.com/ec2/pricing/on-demand/
• https://cloud.google.com/compute/gpus-pricing
• https://mlcommons.org/benchmarks/inference-datacenter/

Related on PULSE

• [How do you choose an inference accelerator: GPU, TPU, or custom silicon?](/knowledge/ai415)
• [What is the difference between vLLM, TGI, and Triton for LLM inference?](/knowledge/ai345)
• [What is the difference between batch and real-time inference infrastructure?](/knowledge/ai409)
• [How do you choose between cloud GPUs and on-prem for AI workloads?](/knowledge/ai375)
• [What is the difference between model parallelism and data parallelism in distributed training in 2027?](/knowledge/ai447)