What is the average cost per million tokens for self-hosted LLM inference compared to API-based inference in 2027?

The two options compared

The comparison in 2027 is no longer "open weights versus closed API." It is a comparison of two cost structures that behave very differently as volume scales. Self-hosted inference converts a variable per-token bill into a fixed capacity bill: you pay for GPUs by the hour whether or not tokens flow through them. API-based inference keeps cost strictly variable, but the provider bakes in a margin, a serving-efficiency advantage, and a risk premium for capacity you did not have to reserve.

Self-hosted inference in 2027 typically means running open-weight models — the Llama, Mistral, Qwen, DeepSeek, and Gemma families — on rented GPU capacity from a hyperscaler or neocloud, or on owned hardware. The dominant deployment pattern is not one GPU per model. It is a shared inference server such as vLLM, TensorRT-LLM, or SGLang, which batches many concurrent requests onto the same GPUs. That batching is what makes the economics work: a single H100 or B200 running continuous batching can serve far more tokens per hour than its raw single-stream throughput suggests.

API-based inference in 2027 means paying per token to a provider — OpenAI, Anthropic, Google, Mistral, Cohere, or a serverless GPU platform like Together, Fireworks, or Bedrock. The headline rates have fallen sharply for small and mid-tier models and have stayed comparatively high for frontier reasoning models. The important structural fact is that API pricing is tiered by capability, not by compute. A small model might sit near $0.10–$0.50 per million tokens, a mid-tier model near $1–$5, and a frontier reasoning model well above $10 per million output tokens.

The comparison therefore has to be made at a matched capability tier. Comparing a self-hosted 8B model against a frontier API model is not a cost comparison; it is a quality comparison. The honest framing is: for a given quality bar, what does each option cost per million tokens, and at what monthly volume does the fixed-cost option cross below the variable-cost option?

Three cost components dominate the self-hosted side and are frequently omitted from naive comparisons:

• GPU rental or amortization. An H100-class instance rents for roughly $2–$4 per GPU-hour in 2027; a B200-class instance for roughly $4–$8. Owned hardware amortizes over 3–4 years but adds power, cooling, and ops.
• Utilization. The single largest swing factor. A GPU at 15% utilization costs roughly six times more per million tokens than the same GPU at 90%.
• Serving overhead. Engineering time for quantization, batching configuration, autoscaling, KV-cache management, and evals. This is real money even when it is not on an invoice.

The API side has three components too, and they are simpler: input token rate, output token rate (usually 3–5x the input rate), and any cached-input discount. Prompt caching in 2027 can cut input costs by 50–90% for workloads with stable prefixes, which materially changes the crossover math for RAG and agent workloads.

The practical conclusion most RevOps and platform teams reach is a hybrid: self-host the high-volume, latency-tolerant, mid-tier workloads; keep API access for frontier reasoning, burst capacity, and low-volume experiments. The rest of this page quantifies where that split should fall.

How to decide between them

The decision is not "which is cheaper" in the abstract. It is "at our projected token volume and utilization, which cost curve are we on?" The workflow below is the one that reliably produces a defensible answer.

The first branch is the one teams skip. Crossover volume in 2027 for a mid-tier model sits somewhere in the range of 300 million to 1.5 billion tokens per month, depending on GPU class, quantization, and how aggressively you batch. Below that, API almost always wins because you are not paying for idle silicon. Above it, self-hosting starts to pull ahead, and the gap widens roughly linearly with volume until you saturate your cluster and have to add more GPUs.

The second branch is the one teams underestimate. Utilization is not a target you set; it is an outcome of traffic shape. A workload with a sharp business-hours peak and near-zero overnight traffic will struggle to clear 40% average utilization no matter how well it is engineered. That workload is a poor self-hosting candidate even at high monthly volume, because the monthly volume is concentrated into too few hours. The fix is either to over-provision for peak (which destroys the unit economics) or to accept queueing and latency degradation at peak (which may be unacceptable).

The third branch — latency and data sensitivity — often overrides pure cost. Regulated workloads, on-premise requirements, and sub-100ms first-token latency targets can force self-hosting regardless of the arithmetic. Conversely, workloads that need frontier reasoning quality cannot be self-hosted at all in most cases, because the best reasoning models are not open-weight.

A useful discipline: build the comparison as a spreadsheet with three columns — API blended cost, self-hosted blended cost at 50% utilization, self-hosted blended cost at 80% utilization — and a row for each model tier you actually use. The decision usually becomes obvious once the utilization sensitivity is visible.

Concrete numbers behind each option

This section gives the arithmetic. All figures are 2027 planning ranges, not quotes; treat them as starting points for your own model.

API-based cost per million tokens. For a mid-tier general-purpose model, input rates in 2027 cluster around $0.50–$2.00 per million input tokens and output rates around $2.00–$8.00 per million output tokens. For a small, fast model, input can be $0.05–$0.30 and output $0.20–$1.00. For frontier reasoning models, input runs $3–$15 and output $15–$60 or more, with reasoning tokens billed as output. Blended cost depends entirely on your input:output ratio. A chat workload at 4:1 input:output on a mid-tier model lands near $1.50–$3.50 per million tokens blended. A summarization workload at 20:1 lands near $0.60–$1.50. An agent workload that generates long reasoning traces can invert the ratio and push blended cost above $5.

Prompt caching changes this materially. If 70% of your input tokens are cache hits at a 75% discount, the effective input rate drops by roughly 52%. On the mid-tier chat example, that pulls blended cost from ~$2.50 down to ~$1.60 per million tokens. Any serious API cost model in 2027 must include a cache-hit-rate assumption.

Self-hosted cost per million tokens. Start with GPU-hour cost. A single H100 80GB instance at $3.00/hour is $2,190 per month if run continuously. A B200-class instance at $6.00/hour is $4,380 per month. Now divide by tokens served.

Throughput is where the estimates diverge. A well-tuned vLLM deployment of a 70B-class model in 4-bit quantization on one H100 can sustain roughly 1,500–3,000 output tokens per second under continuous batching at moderate concurrency, with input prefill adding to the load. Call it 2,000 output tokens/second as a planning midpoint. Over a 730-hour month that is 2,000 × 3,600 × 730 ≈ 5.25 billion output tokens per month at 100% utilization. At 60% utilization, ~3.15 billion. At 30%, ~1.58 billion.

Now the arithmetic:

• 100% utilization, $2,190/month, 5.25B tokens → ~$0.00042 per 1,000 tokens, or ~$0.42 per million tokens.
• 60% utilization → ~$0.70 per million tokens.
• 30% utilization → ~$1.39 per million tokens.
• 15% utilization → ~$2.78 per million tokens.

That is the whole story of self-hosted inference economics in one table. The same GPU, the same model, the same software stack, and a 6.6x spread in unit cost driven purely by how busy the hardware is.

Smaller models shift the curve favorably. An 8B model in 4-bit on the same H100 can push 8,000–15,000 output tokens/second, which at 60% utilization yields roughly $0.10–$0.18 per million tokens. That is genuinely below what most APIs charge for equivalent small-model quality. It is also the segment where self-hosting most often wins.

Larger models shift it unfavorably. A 400B+ open-weight model may need 4–8 GPUs per replica, multiplying the hourly cost while throughput scales sub-linearly. At that point self-hosted cost per million tokens can exceed $2–$5 even at good utilization, and the comparison against a mid-tier API becomes uncomfortable.

The crossover. Set API blended cost equal to self-hosted blended cost and solve for volume. For a mid-tier model at $2.00 per million tokens API versus a self-hosted cost of $2,190/month fixed plus a small marginal cost, the crossover is roughly 1.1 billion tokens per month at 60% utilization. Below that, API wins. Above it, self-hosting wins and the advantage grows. For a small model at $0.30 per million API versus $2,190/month fixed, crossover is ~7.3 billion tokens per month — which is why most teams do not self-host small models unless they already have the GPUs.

Hidden costs on the self-hosted side. Add these before declaring victory:

• Engineering time. Expect 0.5–2 FTE for a production inference stack, including upgrades, quantization tuning, autoscaling, and incident response. At a fully loaded $200k/year, that is $100k–$400k annually — equivalent to 45–180 billion tokens of API spend at $2 per million. This single line item sinks many self-hosting business cases at moderate volume.
• Idle and failover capacity. Production needs headroom for failover and peak. Realistic average utilization for a two-region deployment is often 35–50%, not 80%.
• Egress and storage. Model weights, logs, and eval data add modest but non-zero cost.
• Quality gap. If the open-weight model is 5% worse on your task, and that 5% costs you conversion or support volume, the effective cost is higher than the invoice suggests.

Hidden costs on the API side. Rate limits during peak, provider deprecations that force migrations, and the fact that you cannot negotiate meaningful volume discounts below very large commitments. Also: token accounting is not always intuitive, and reasoning models can consume far more output tokens than expected for the same user-visible answer.

A worked hybrid example. Suppose a company processes 900 million tokens per month: 700 million on a mid-tier model and 200 million on a frontier reasoning model. API cost at $2.00 blended for the mid-tier and $12.00 blended for the frontier is $1,400 + $2,400 = $3,800/month. Self-hosting the mid-tier on one H100 at 50% utilization costs ~$2,190/month in GPU plus an allocated ~$8,000/month in engineering — clearly worse. But if volume triples to 2.1 billion mid-tier tokens, self-hosted GPU cost stays at $2,190–$4,380 while API cost rises to $4,200, and the engineering allocation is amortized across three times the volume. That is the shape of the decision: it is a volume story, not a technology story.

Implementation details and sequencing

If the numbers say self-host, the next question is how to get there without breaking production. The sequencing below is the pattern that works.

Step 1: Instrument before you migrate. You cannot compare costs without knowing your actual token mix. Break usage down by model, by route, by tenant, and by input:output ratio. Most teams discover that a small number of routes account for the majority of tokens — often a single summarization or classification endpoint. Those are the candidates.

Step 2: Choose the model and quantization deliberately. Match the open-weight model to the quality bar of the API model you are replacing, not to the largest model you can fit. 4-bit quantization (AWQ, GPTQ, or FP8 on newer hardware) typically preserves quality within a small margin while roughly doubling throughput and halving memory. Test on your own eval set; public benchmarks are a weak proxy.

Step 3: Pick the serving engine and configure batching. vLLM with continuous batching and paged attention is the default in 2027. TensorRT-LLM wins on raw throughput for NVIDIA hardware when you are willing to invest in compilation. SGLang is strong for structured and prefix-heavy workloads. The configuration knobs that matter most are max batch size, max sequence length, GPU memory utilization fraction, and whether prefix caching is enabled. Prefix caching is close to free money for RAG workloads with stable system prompts.

Step 4: Deploy one replica and shadow-test. Run the self-hosted model alongside the API on sampled production traffic, compare outputs, and measure quality deltas on your own metrics. Do not skip this. Quality regressions are the most common reason a cost-driven migration gets rolled back.

Step 5: Shift traffic gradually. Move 5–10% of production traffic, then measure p50 and p99 latency, throughput, and error rate under real load. Increase in steps. Keep the API path warm as a fallback and as burst capacity.

Step 6: Build the autoscaling and failover story. Self-hosted inference needs a scaling policy tied to queue depth, not CPU. Scale on pending requests and KV-cache pressure. Have a documented failover to API when the cluster is saturated or unhealthy — this is what makes the hybrid model robust.

Step 7: Re-measure unit cost monthly. Track actual GPU hours, actual tokens served, and actual utilization. The number that matters is realized cost per million tokens, not the theoretical one. Utilization drifts, model versions change, and traffic mix shifts. A quarterly re-run of the crossover analysis keeps the decision honest.

Sequencing pitfalls. Do not migrate the highest-stakes route first. Do not assume peak capacity equals average capacity. Do not forget the engineering allocation. And do not treat the API as a temporary crutch — in most mature 2027 stacks, the API remains a permanent part of the architecture for frontier reasoning, burst, and failover.

Related questions

Does self-hosted inference get cheaper every year?

Yes, but slower than API prices fall. Hardware efficiency improves and quantization gets better, but GPU rental rates and engineering salaries do not collapse. The crossover volume tends to drift upward, not downward, for mid-tier models.

What utilization rate do I need for self-hosting to beat API pricing?

For a mid-tier model, roughly 50–60% sustained utilization at volumes above about one billion tokens per month. Below 40%, API pricing almost always wins once engineering time is counted.

Is prompt caching relevant to the comparison?

Yes, heavily. Caching can cut API input costs by 50–90% on stable-prefix workloads, which raises the crossover volume and makes API relatively more attractive for RAG and agent patterns.

Can I self-host frontier reasoning models?

Generally no. The strongest reasoning models in 2027 are not open-weight. Open-weight reasoning models exist and are improving, but they trail the frontier, so the comparison is not like-for-like.

How much engineering time does self-hosted inference really take?

Plan for 0.5–2 FTE for a production deployment, more during initial build. That is often the largest single line item and the one most frequently omitted from cost comparisons.

FAQ

What is the average cost per million tokens for self-hosted LLM inference in 2027?

At 60% utilization on an H100-class GPU, a 70B model in 4-bit quantization lands around $0.60–$0.90 per million output tokens. A small 8B model on the same hardware lands near $0.10–$0.20. At low utilization, the same deployments can exceed $2.00 per million tokens. There is no single average — utilization is the dominant variable.

How does that compare to API-based inference in 2027?

Mid-tier API models blend to roughly $1.00–$3.50 per million tokens depending on input:output ratio and cache hit rate. Small-model APIs sit near $0.10–$0.60. Frontier reasoning APIs run $10–$60 or more per million output tokens. So self-hosted mid-tier is often cheaper at scale, while API is cheaper at low volume and for frontier quality.

What is the crossover volume where self-hosting becomes cheaper?

For a mid-tier model, roughly 300 million to 1.5 billion tokens per month at 50–60% utilization. For small models, the crossover is much higher — often above 5 billion tokens per month — because API small-model rates are already very low.

Does the comparison change for input-heavy versus output-heavy workloads?

Yes, significantly. Output tokens dominate GPU compute, so output-heavy workloads make self-hosting look better relative to API output rates, which are typically 3–5x input rates. Input-heavy workloads with high cache-hit rates favor API, because caching discounts apply to input.

Should I count engineering time in the comparison?

Always. A production inference stack needs ongoing engineering for upgrades, tuning, autoscaling, and incidents. At 0.5–2 FTE, that cost can exceed the GPU bill and is the most common reason a self-hosting business case fails at moderate volume.

Is a hybrid approach the right answer?

For most organizations above a few hundred million tokens per month, yes. Self-host the high-volume, latency-tolerant baseline; keep API access for frontier reasoning, burst capacity, and failover. This captures most of the cost benefit without sacrificing quality or resilience.

Sources

• https://docs.vllm.ai/en/latest/
• https://huggingface.co/docs/transformers/quantization
• https://openai.com/api/pricing/
• https://www.anthropic.com/pricing
• https://cloud.google.com/vertex-ai/generative-ai/pricing
• https://aws.amazon.com/bedrock/pricing/
• https://www.nvidia.com/en-us/data-center/h100/
• https://arxiv.org/abs/2309.06180

Related on PULSE

• How to model token costs across API and self-hosted inference
• GPU utilization benchmarks for production LLM serving
• Quantization trade-offs for cost and quality
• Prompt caching strategies for RAG and agent workloads
• Hybrid inference architectures: routing between API and self-hosted
• Build-versus-buy frameworks for AI infrastructure