How do you choose an inference accelerator: GPU, TPU, or custom silicon?

The outcome you should expect

The realistic outcome of a well-run accelerator selection is not a headline speedup. It is a shift in your unit economics and your ability to make latency promises you can keep. Teams that pick correctly typically see cost per million tokens fall somewhere between 30% and 60% versus their naive first deployment, and — just as important — they see p99 latency stop wandering. The second effect is usually worth more than the first, because a stable p99 is what lets a product team ship a synchronous feature instead of hiding inference behind a spinner or a job queue.

Be honest about where the gains actually come from. In most audits of a first-generation deployment, the accelerator itself is not the largest lever. Serving software (continuous batching, paged attention, quantization), request shaping, and model right-sizing usually account for more of the improvement than the silicon swap does. A team running an unbatched FP16 7B model on an H100 is leaving more on the table from its serving stack than it would gain by moving to a different vendor. So the correct expectation is: fix the serving layer first, then choose hardware against the workload you actually have, and expect the hardware decision to be worth a solid but not miraculous fraction of the total.

The other outcome you should expect is a constraint you did not have before. Every accelerator choice buys performance with portability. A CUDA deployment runs nearly anywhere NVIDIA silicon exists — cloud, colo, workstation, three vendors deep. A TPU deployment runs on Google Cloud, full stop, and is written in JAX or TensorFlow with XLA doing the heavy lifting. Custom inference silicon such as Groq's LPU, AWS Inferentia, or Cerebras' wafer-scale system gives you a compiler and a supported-operator list, and anything outside that list is your problem to solve or your reason to fall back. Going in, decide how much of that portability you are willing to trade, because you will trade some.

Finally, expect the decision to have a shelf life. Accelerator generations turn over roughly every 18 to 24 months, and cloud pricing for the prior generation typically drops meaningfully once the successor ships in volume. That means the right frame is not "which chip is best" but "which chip is best for the next four to six quarters of my workload, at a commitment level I can unwind." A three-year on-premises purchase and a one-month cloud reservation are different bets on the same question, and they deserve different amounts of certainty about your traffic.

What drives that outcome

Four variables drive almost every inference accelerator decision, and they dominate everything else on the spec sheet.

Model size versus memory capacity. This is the first and hardest gate. A model must fit — weights, KV cache, activation scratch — in the memory attached to the accelerator, or you pay a parallelism tax. Rough arithmetic: weights consume about 2 bytes per parameter at FP16/BF16, about 1 byte at FP8/INT8, and roughly 0.5 bytes at 4-bit. So a 70B model is roughly 140 GB at FP16, ~70 GB at 8-bit, ~35 GB at 4-bit. Then add KV cache, which scales with batch size times context length and can easily rival the weights on long-context workloads. If your model plus cache fits on one device, you avoid cross-device communication entirely and your life gets much simpler.

Arithmetic intensity — are you compute-bound or bandwidth-bound? Transformer decoding is famously memory-bandwidth-bound at small batch sizes: each generated token requires streaming the full weight set through the compute units, so tokens per second tracks HBM bandwidth far more closely than it tracks peak TFLOPS. Prefill, by contrast, is compute-bound and does reward raw FLOPS. This split explains why architectures with enormous on-chip SRAM (Groq's LPU, Cerebras' wafer-scale engine) post remarkable single-stream latency despite modest headline math throughput — they sidestep the bandwidth wall rather than climbing it. If your workload is single-stream chat, weigh bandwidth and memory architecture. If it is bulk document processing with large batches, weigh FLOPS and interconnect.

Software ecosystem and operator coverage. CUDA plus TensorRT-LLM, Triton, and vLLM is the deepest, most forgiving stack; almost any model architecture off Hugging Face runs the day it lands. AMD's ROCm has closed a lot of that gap for mainstream transformer serving via PyTorch and vLLM, but exotic kernels and brand-new architectures still show up there later. XLA-compiled TPU workloads are excellent for well-trodden shapes and painful for dynamic control flow. Custom silicon compilers — AWS Neuron, Habana's SynapseAI, Groq's compiler — are generally fine on the models their vendor markets and unpredictable outside them. Budget engineering weeks accordingly; a 20% cheaper chip that costs you a quarter of porting work is not cheaper.

Deployment locus and commitment shape. On-premises makes sense at high sustained utilization; cloud makes sense at variable or unproven load. The crossover is basically a utilization question. Amortize a card over three years, add power and cooling (a modern data-center accelerator draws several hundred watts, and facility overhead adds 20–50% on top), and compare against the on-demand hourly rate at your real duty cycle. Teams running below roughly 40–50% utilization almost always lose money buying hardware; teams running pinned at 80%+ almost always win.

Benchmarks and realistic ranges

Treat every vendor number as an upper bound achieved under conditions you will not reproduce. The useful discipline is to convert marketing figures into ranges you can plan against, then validate with your own traffic.

On MLPerf. MLCommons' MLPerf Inference suite is the only broadly comparable public benchmark, and it is worth reading — but read it carefully. Results are split into Closed (fixed model, apples-to-apples) and Open (vendor may modify the model) divisions, and into Server (latency-constrained, arrival distribution) and Offline (maximum throughput) scenarios. A number pulled from Offline Open and quoted next to a Server Closed number is not a comparison, it is a category error. Also note who submitted: many rounds have far more entries from one vendor's partners than from others, so absence from the results table is not evidence of poor performance.

Precision and where the throughput actually comes from. Modern accelerators advertise FP8 or INT8 tensor throughput, and sparse figures that assume 2:4 structured sparsity are frequently double the dense number. Always check whether a quoted TFLOPS figure is dense or sparse. In practice, moving a transformer from FP16 to 8-bit weight-and-activation quantization tends to deliver something in the neighborhood of a 1.5–2× throughput improvement and roughly halves the memory footprint, with small accuracy deltas on most tasks when calibrated properly. Four-bit weight-only quantization halves memory again and helps a lot on bandwidth-bound decoding, but accuracy sensitivity rises and varies significantly by model and task. Measure on your evals, not on a leaderboard.

Realistic planning ranges. Rather than quoting exact tokens per second, anchor on relationships that hold across generations. Single-stream decode throughput for a given model scales roughly with memory bandwidth. Batched throughput scales roughly with usable compute until you hit a memory wall. Doubling batch size typically increases throughput sub-linearly while increasing p99 latency super-linearly — this is the core tension of serving, and the reason continuous batching (as in vLLM, TensorRT-LLM's in-flight batching, and similar) is worth more than most hardware upgrades. Expect a well-tuned continuous-batching server to deliver several times the throughput of a naive static-batch server on identical silicon.

Cost framing. Compare on cost per million tokens at your target latency, never on cost per hour or cost per card. The conversion is straightforward: measured tokens per second at your accepted p99, times 3,600, divided into your all-in hourly cost. Do that for each candidate and the ranking often inverts relative to the spec-sheet ordering, because a cheaper chip that hits 70% of the throughput at 50% of the price wins on the metric that matters. Include the parts people forget: idle capacity during off-peak hours, the replica count you need for redundancy and rolling deploys, egress, and the engineer-months of porting.

Benchmark your own traffic. The single highest-value step is replaying a captured sample of real production requests — real prompt-length distribution, real output lengths, real concurrency pattern — against each candidate. Synthetic benchmarks with uniform 128-token prompts tell you almost nothing about a workload whose prompts range from 200 to 20,000 tokens. Capture a week of request shapes, build a load generator that reproduces the arrival distribution, and measure p50, p95, p99, and cost per million tokens for each option. Two days of that work routinely changes the decision.

Risks, edge cases, and failure modes

Buying for peak instead of steady state. The most common and most expensive error is sizing a fleet for the highest hour of the highest day and then running it at 25% for the rest of the month. Inference load is usually spiky and often business-hours-shaped. The fix is architectural rather than procurement: keep a reserved or owned base layer sized to the sustained floor, and burst the peak into on-demand cloud capacity or a smaller, cheaper accelerator tier. Mixed fleets are more operational work but dramatically better on cost.

Underestimating the port. Every non-CUDA path has an operator coverage boundary. You will find it when a model update introduces a new attention variant, a custom kernel, or a sampling method the compiler does not support. The failure mode is not a crash — it is a silent fallback to a slower path, or a two-week delay shipping the model everyone else already has. Mitigate by pinning an explicit list of the model families you must support over the next year, testing each on the candidate stack *before* committing, and keeping a GPU-based escape hatch capable of carrying production traffic at reduced efficiency.

Numerical drift after quantization. Aggressive quantization can pass aggregate accuracy checks while degrading specific behaviors — long-context recall, structured output validity, tool-call formatting, non-English performance, safety-relevant refusals. Aggregate benchmark parity is not the same as behavioral parity. Build a task-specific eval that mirrors your actual product surface, run it at every precision, and gate promotion on it. This is where a hardware migration most often causes a quiet quality regression that shows up weeks later as user complaints.

Memory fragmentation and KV cache blowups. Long-context requests can consume KV cache far out of proportion to their frequency. A handful of 100k-token requests can evict everything else and collapse throughput for the whole node. Paged-attention style KV management mitigates this, but you still need admission control: cap max context per tier, route long-context requests to a dedicated pool with more memory per device, and alarm on cache-eviction rate as a first-class metric alongside latency.

Supply, lead times, and regional capacity. Accelerator availability is not uniform. Popular parts have had multi-month lead times, and cloud capacity for the newest instance families is frequently constrained to a handful of regions, which interacts badly with data-residency requirements. If your architecture assumes a specific chip in a specific region, verify capacity before you design around it — and confirm whether your provider will actually sell you a reservation at the size you need.

Interconnect as the hidden bottleneck. Once a model spans devices, the fabric matters as much as the chip. Tensor parallelism is chatty and needs high-bandwidth, low-latency links; running it across a commodity network instead of a proper accelerator fabric can erase the benefit of the faster chip entirely. Check topology, not just chip count: eight accelerators in one coherent domain behave very differently from eight spread across two boxes.

Vendor concentration risk. Standardizing on one accelerator gives you the best per-unit efficiency and the worst negotiating position. The pragmatic middle ground is an abstraction layer at the serving API — keep model definitions in PyTorch, keep the serving interface OpenAI-compatible or equivalent, and treat the backend as swappable. You will lose a few points of peak efficiency and gain the option to move, which has repeatedly been worth more when pricing or supply shifts.

Edge and on-device as a real alternative. For privacy-sensitive or intermittently-connected workloads, the right accelerator may not be in a data center at all. Unified-memory workstation silicon and mobile NPUs now run quantized small models locally at low power. The economics are entirely different — capex per seat rather than opex per token — and for a subset of use cases (regulated document handling, offline field tools) it removes an entire class of compliance work.

A practical rollout plan

Run the selection as a time-boxed evaluation, not an open-ended research project. Four to six weeks is a reasonable envelope for a serious decision.

Week 1 — instrument and capture. Before touching hardware, get your current workload measured. Log prompt-token and completion-token distributions, arrival rate by hour, concurrency, and current p50/p95/p99. Compute your current cost per million tokens honestly, including idle capacity. Most teams discover something surprising here: that median prompts are far longer than assumed, that 5% of requests consume 40% of compute, or that utilization is half what they believed. Everything downstream depends on this baseline being real.

Week 2 — fix the serving layer on existing hardware. Enable continuous batching, paged KV cache, and an appropriate quantization level on whatever you already run. Re-measure. This step frequently produces a large improvement at zero procurement cost, and it establishes a fair baseline — otherwise you will credit a hardware change for gains that came from software you could have had anyway.

Week 3 — shortlist and smoke-test. Pick two or three candidates that survive the hard gates: memory capacity versus model size, framework and operator coverage for your required model families, and regional/commercial availability. Get short-term cloud access to each — nearly every accelerator family is rentable by the hour, which makes this cheap. Port one representative model to each and confirm it runs correctly at your target precision before measuring anything.

Week 4 — replay production traffic. Run your captured request distribution against each candidate at several concurrency levels. Record tokens per second at your p99 ceiling, cost per million tokens, and quality eval scores at each precision. Also record the un-glamorous things: how long the port took, what broke, how good the error messages were, whether the profiler told you anything useful. Operational ergonomics compound over years.

Weeks 5–6 — shadow, then canary. Mirror a slice of live traffic to the winning candidate and compare outputs against production for both latency and quality drift. Then route real traffic behind a feature flag — 1%, then 5%, then 25% — with automatic rollback on latency or error-rate regression. Keep the incumbent warm through the full ramp. Only after a clean week at 25% should you consider a purchase or reservation commitment.

Ongoing — re-evaluate on a schedule. Put a recurring review on the calendar every two quarters. Check whether a new generation has shipped, whether last-generation pricing has dropped, and whether your workload mix has shifted (context lengths creep upward; batch profiles change as products mature). Keep the traffic-replay harness you built — it is the durable asset from this whole exercise, and it turns each future evaluation from a month into a few days.

Related questions

Does the accelerator choice affect revenue directly?

Indirectly but measurably. Lower cost per token widens gross margin on any AI feature you meter, and a stable p99 lets product teams ship synchronous experiences that convert better than queued ones. Model the decision as margin plus conversion impact, not just an infrastructure line item.

Should a small team even consider anything other than GPUs?

Usually no. Below roughly a few hundred thousand dollars of annual inference spend, the engineering cost of porting to alternative silicon outweighs the savings. Use managed GPU capacity or a hosted API, invest in serving-layer tuning, and revisit alternatives once volume is predictable and large.

How does training hardware choice interact with inference?

Less than people assume. Training and inference have different bottlenecks, and it is entirely normal to train on one platform and serve on another since weights are portable. The main coupling is the framework: models authored in JAX are easier to serve on TPUs, PyTorch models on GPUs.

What about running inference on CPUs?

Viable for small models, low request volume, or embedded scenarios, especially with modern CPU matrix extensions and aggressive quantization. For anything transformer-shaped above a few billion parameters at interactive latency, CPUs lose decisively on memory bandwidth. Use them for classical ML, embeddings at low volume, and fallback capacity.

How do I compare a hosted API against buying accelerators?

Convert both to cost per million tokens at equal quality and latency. Hosted APIs bundle capacity risk, upgrades, and operations into the price, which is genuinely valuable below high sustained volume. Self-hosting wins when utilization is high, the model is fixed, and data residency or customization requires it.

FAQ

What is the practical difference between a GPU, a TPU, and custom inference silicon?

A GPU is a general-purpose parallel processor with tensor units bolted on and an enormous software ecosystem around it — it runs whatever you throw at it. A TPU is an ASIC built around large systolic matrix units and a compiler-first workflow, excellent on stable transformer shapes and awkward on dynamic control flow. Custom inference silicon spans a wide range, from cloud-vendor inference chips optimized for cost per token to architectures that trade external memory for enormous on-chip SRAM to win on deterministic latency. The general rule: as you move from GPU toward custom silicon, you gain efficiency on the workloads the vendor targeted and lose the ability to run anything else.

Should I optimize for latency or throughput?

Decide based on where the request sits in the user's experience. Anything in a synchronous path — chat, autocomplete, voice, in-product assistance — is latency-governed, and you should optimize p99 at modest batch sizes; there, memory bandwidth and single-stream behavior matter most. Anything asynchronous — bulk classification, document enrichment, nightly summarization, embedding backfills — is throughput-governed, and you should maximize tokens per dollar at whatever batch size the memory allows. Most organizations have both, which is a strong argument for a two-tier fleet rather than one accelerator type for everything.

How much does quantization change the hardware decision?

Substantially, because it moves the memory gate. A model that needs multiple devices at FP16 may fit on one at 8-bit and comfortably at 4-bit, and single-device serving eliminates parallelism overhead entirely. That can make a mid-tier accelerator outperform a flagship on cost per token for your specific model. The caveat is that quantization must be validated on your own task evals, not just on public benchmarks — degradation tends to be uneven, hitting long-context and structured-output behaviors harder than aggregate scores suggest.

Is it worth running a mixed fleet of different accelerators?

Yes if you have genuinely distinct workload tiers and enough platform engineering capacity to maintain more than one serving path. A common pattern is flagship GPUs for the newest and largest models, cheaper GPUs or vendor inference chips for high-volume steady workloads, and a burst tier on on-demand cloud. The cost is real: two stacks means two sets of deployment tooling, monitoring, and on-call knowledge. Below a certain scale, homogeneity is worth more than the efficiency you would gain.

How often should I revisit this decision?

Every two quarters is a sensible cadence, with an out-of-band review whenever a new accelerator generation ships or your workload mix shifts materially. New generations reset the price-performance curve and, just as usefully, push prior-generation capacity down in price — which is often the better buy for steady inference. Keep the traffic-replay harness from your original evaluation so each revisit costs days rather than weeks.

What single metric should drive the final call?

Cost per million tokens at your committed p99 latency, measured on replayed production traffic, with quality evals held constant across candidates. That one number folds together throughput, memory efficiency, precision choice, and price. Everything else — peak TFLOPS, memory bandwidth, chip count — is an input to it rather than a decision criterion on its own. If a candidate cannot be measured that way because it will not run your model, that is your answer.

Sources

• MLPerf Inference Benchmark Results — MLCommons
• NVIDIA H100 Tensor Core GPU
• Google Cloud TPU Documentation
• AWS Inferentia
• AMD Instinct Accelerators
• Intel Gaudi AI Accelerators
• vLLM Documentation
• NVIDIA TensorRT-LLM
• OpenXLA Project
• Groq

Related on PULSE

• [How do you choose between cloud GPUs and on-prem for AI workloads?](/knowledge/ai375)
• [What is GPU memory fragmentation and how do you avoid it?](/knowledge/ai397)
• [How do you measure and improve GPU utilization?](/knowledge/ai433)
• [The 10 Best GPU Orchestration Tools for Kubernetes in 2027](/knowledge/ai358)
• [How do you choose a vector database for a production RAG system in 2027?](/knowledge/ai339)