How do you build a self-hosted LLM stack in 2027?

The outcome you should expect

The realistic outcome of a self-hosted build is not "cheaper than an API on day one." It is predictable unit economics, hard data residency, and freedom from per-token metering — bought with capital expenditure and engineering time you would otherwise not spend.

Expect three concrete deliverables from a first build. First, a running inference endpoint that speaks the OpenAI chat-completions schema, so existing application code changes only its base URL and API key. Second, a model artifact you control — downloaded weights sitting on your own disk, versioned and reproducible, that no upstream provider can deprecate out from under you. Third, a cost curve that is flat with respect to volume: once the hardware is paid for, the marginal cost of a token is electricity and amortization, not a line item that scales with usage.

Expect the following not to happen. You will not match a frontier hosted model's quality on the hardest reasoning tasks — open-weight models have closed much of that gap on summarization, extraction, classification, and routine drafting, but the top of the capability curve still belongs to the large proprietary systems. You will not get zero operational burden; someone owns patching, driver upgrades, model refreshes, and the pager. And you will not see a sensible payback period at low volume. If your token spend is a few hundred dollars a month, self-hosting is a data-control decision or a learning project, not a savings decision.

The sane way to frame the outcome is a break-even calculation. Take your current monthly inference spend, subtract the estimated electricity and hosting cost of the equivalent on-prem capacity, and divide the hardware capital cost by the difference. Add loaded engineering hours for setup and ongoing maintenance — realistically a meaningful fraction of one engineer's time in year one. If that break-even lands beyond roughly 18-24 months, the economics alone do not justify the build; you need a second reason, and the usual second reason is that your data cannot leave your perimeter.

The strongest cases for self-hosting are: regulated data that contractually cannot transit a third party; workloads with high, steady, predictable volume where per-token pricing compounds badly; air-gapped or intermittently-connected environments; and latency-sensitive paths where a round trip to a public endpoint is too slow or too variable. The weakest cases are spiky, low-volume, quality-maximizing workloads — those belong on a hosted API.

What drives that outcome

Four variables dominate everything else: model size, quantization, memory bandwidth, and batching. Understanding how they interact is the difference between a stack that hits its numbers and one that quietly underperforms by an order of magnitude.

Model size sets the floor. Weights in 16-bit precision consume roughly two bytes per parameter, so a 70-billion-parameter model needs about 140GB just to hold weights, before any key-value cache. That single fact determines your hardware class. Quantize to 8-bit and you land near 70GB; quantize to 4-bit and you land near 35-40GB, which fits on a single high-VRAM professional card. This is why 4-bit quantized 70B on one GPU is the canonical single-workstation configuration — it is the largest model that fits in the most affordable serious card.

Quantization trades quality for capacity, non-linearly. Going from 16-bit to 8-bit is nearly free in output quality for most business tasks. Going from 8-bit to 4-bit is usually acceptable and is where most self-hosters live. Below 4-bit, degradation becomes visible on reasoning-heavy and long-context tasks, and it shows up unevenly — a model may still write fluent prose while silently getting arithmetic and structured extraction wrong. Never ship a sub-4-bit quantization without running your own evaluation set against the full-precision baseline.

Memory bandwidth, not raw compute, governs single-stream speed. Autoregressive decoding reads the entire weight set from memory for every token generated. A card's tokens-per-second ceiling for batch size one is approximately its memory bandwidth divided by the model's on-disk size in bytes. This is why a consumer card with high bandwidth can feel fast on a small model, and why unified-memory systems with large capacity but modest bandwidth run big models at usable-but-not-fast speeds. It is also why adding GPUs speeds up capacity far more than it speeds up any single user's stream.

Batching is where throughput actually comes from. Serving one request at a time wastes the vast majority of a datacenter GPU's compute. Continuous batching — the technique at the heart of modern production inference engines — interleaves many in-flight requests, so aggregate throughput can be roughly an order of magnitude higher than single-stream speed while per-user latency stays acceptable. Paged key-value cache management, which allocates cache in fixed blocks rather than contiguous per-request slabs, is what makes high batch sizes possible without fragmenting memory. If your workload is many concurrent short requests, a batching engine is the single highest-leverage component in the stack. If your workload is one developer chatting, batching buys you nothing and you should optimize for setup simplicity instead.

The secondary drivers matter less individually but compound. Context length inflates key-value cache linearly, and long-context workloads can consume as much memory in cache as the weights themselves — a 32K-token conversation on a large model is a materially different memory profile than a 2K-token one. Prompt caching, where the engine reuses computed attention state for a shared system prompt across requests, meaningfully cuts time-to-first-token for repetitive workloads. Speculative decoding, where a small draft model proposes tokens that the large model verifies in parallel, can substantially improve throughput on predictable text at the cost of extra memory for the draft model.

Benchmarks and realistic ranges

Publish nothing you have not measured on your own hardware with your own prompts. Vendor and community benchmark numbers are directionally useful and almost always optimistic, because they are measured with short prompts, ideal batch composition, and warm caches. That said, here are the ranges that hold up as planning assumptions.

Single-user workstation, 4-bit 70B on one 48GB professional card. Expect tens of tokens per second for a single stream — fast enough to read along comfortably, slow enough that a long generation takes real time. Time-to-first-token on a short prompt should be well under a second. Concurrency is severely limited: two or three simultaneous users will each see their throughput drop roughly proportionally, because a single card serving a large model at 4-bit has little headroom for batching once key-value cache is accounted for.

Single-user workstation, 4-bit 7B-8B class model on a 24GB consumer card. Expect a multiple of the 70B speed — often 2-3x faster per stream — and enough spare memory for meaningful batching. This is the configuration where a self-hosted stack genuinely feels snappy, and it is the right starting point for classification, extraction, routing, and summarization workloads where a small model is sufficient.

Multi-GPU server with a production batching engine. Aggregate throughput scales roughly with GPU count for capacity, not for single-stream latency. A four-GPU node running a large model under continuous batching can serve many concurrent users at aggregate token rates an order of magnitude above the single-card case, while each individual user sees per-stream speed similar to or modestly better than the single-card number. Plan capacity by aggregate tokens per second and target concurrency, never by single-stream speed multiplied by user count.

CPU-only inference. Viable for small models and batch jobs, not for interactive use. Expect single-digit tokens per second on a modern many-core desktop CPU running a 7B-8B class model with aggressive quantization. This is genuinely useful for overnight batch extraction over a document corpus; it is not useful for a chat interface.

Apple Silicon and unified memory. Large unified memory pools let a single machine hold a 70B model without any discrete GPU, which is a real capability advantage. Throughput sits below a comparable discrete-GPU setup because memory bandwidth is the binding constraint, but power draw and noise are dramatically lower. For a single developer who already owns the machine, this is often the best-value path to running a large model locally.

Operating cost. A single-GPU workstation under sustained load draws in the low hundreds of watts; a dense multi-GPU server draws in the low thousands. At typical commercial electricity rates, the workstation costs tens of dollars a month to run continuously and the server costs low hundreds — before cooling, which in a real rack adds a meaningful multiplier. Electricity is rarely the deciding factor; capital cost and engineering time are.

Measure these five numbers on your own build, and re-measure after every model or driver change: tokens per second at batch size 1, aggregate tokens per second at your target concurrency, time-to-first-token at the 50th and 95th percentile, peak GPU memory at your longest supported context, and requests per second before latency degrades past your service level objective. Everything else is commentary.

Risks, edge cases, and failure modes

Out-of-memory under long context is the most common production failure. Your model fits, your test prompts fit, and then a user pastes a long document and the key-value cache pushes you past VRAM. The fix is to set a hard maximum context length in the engine configuration and reject or truncate over-length requests explicitly, rather than letting the allocator fail mid-generation. Test at your declared maximum context with your declared maximum concurrency simultaneously — the failure only appears when both are at their limits.

Quality regression on quantization is silent. A 4-bit model produces fluent, confident, wrong output. It does not throw an error. Before and after any quantization change, run a fixed evaluation set of a few hundred representative prompts with known-good answers and compare. Structured extraction and numerical reasoning degrade first and most visibly; freeform prose degrades last, which is exactly why casual eyeballing misses the problem.

Driver, CUDA, and engine version coupling breaks upgrades. Inference engines pin narrow ranges of GPU driver and CUDA toolkit versions, and a routine host OS update can silently break a working stack. Pin everything: container image digests, engine versions, driver versions. Keep a known-good image tagged and tested so a rollback is one command. Never upgrade the driver and the engine in the same change window.

Single-node deployment has no failover. One machine is one outage. If the stack is on a revenue-generating path — a customer-facing assistant, an in-product feature — a single node is an availability risk that will eventually cost more than the hardware you saved. The mitigation is either a second node behind a load balancer or a documented, tested fallback to a hosted API when the local endpoint fails health checks. Build the fallback path first, on day one, before anyone depends on the endpoint.

Model licenses are not uniformly permissive. "Open weights" is not the same as "open source," and several widely-used open-weight models carry usage restrictions, acceptable-use policies, or thresholds above which commercial use requires a separate agreement. Read the actual license for every model you deploy, and record which license governs which artifact. This is a legal review item, not an engineering preference.

Security surface is broader than teams expect. A self-hosted endpoint with no authentication on an internal network is an open door. Require API keys or mutual TLS, put the endpoint behind a reverse proxy, rate-limit per client, and never expose the inference port directly to the internet. Prompt injection risk does not decrease because the model is local — if the model has tool access or reads untrusted documents, the same input-sanitization and least-privilege rules apply.

Thermal and power planning gets skipped. High-end GPUs need real power headroom and real airflow. A card that thermally throttles loses a substantial fraction of its throughput, and the symptom looks like "the model got slower" rather than "the cooling is inadequate." Verify power supply capacity against sustained draw plus margin, confirm PCIe slot generation and lane allocation, and monitor GPU temperature and clock throttling as first-class metrics alongside latency.

Staffing is the underrated risk. The stack is easy to stand up and steady to maintain, but it must have a named owner. Builds that fail usually fail because the person who set it up moved on and nobody else knew how the model was updated or where the configuration lived. Document the runbook before the endpoint goes into anything that matters.

A practical rollout plan

Roll out in stages, and make each stage prove its value before the next one gets funded. Do not buy the multi-GPU server first.

Stage one: prove the workload on borrowed hardware. Before any purchase, run your actual prompts against the candidate open-weight model on a rented GPU instance for a few days. The question you are answering is narrow: is the quality good enough for this specific task? Build the evaluation set now — a few hundred real prompts with acceptable and unacceptable outputs marked — because you will use it at every subsequent stage. If quality fails here, no amount of hardware fixes it, and you have spent almost nothing to learn that.

Stage two: single workstation, single user. Buy or repurpose one machine with a high-VRAM GPU. Install a simple local runner and a web frontend, pull a quantized model, and use it daily. The goal is operational familiarity — how models are updated, how long downloads take, what breaks. Keep it off the critical path entirely. Two to four weeks here surfaces most of the surprises.

Stage three: an API endpoint behind a proxy. Replace the desktop-oriented runner with a production inference engine exposing an OpenAI-compatible endpoint. Put a reverse proxy in front with authentication, rate limiting, and request logging. Point one non-critical internal tool at it. Now you are measuring the five numbers from the benchmarking section under something resembling real traffic.

Stage four: retrieval and structure. Add a vector database and an embedding model for retrieval over your own documents, and add structured-output enforcement so downstream code can parse responses reliably. This is where a self-hosted build starts creating value a generic hosted endpoint cannot, because the retrieval corpus is proprietary data that never leaves your network.

Stage five: scale and harden. Only after stages one through four have held up under real traffic do you size the production node. By then you have measured aggregate throughput requirements, so you can specify GPU count against a real number rather than a guess. Add the second node or the hosted-API fallback, wire metrics into your existing observability system, and write the runbook.

Two governance rules make the rollout stick. First, every stage gate is a measurement, not an opinion — you advance when the numbers clear a threshold you wrote down beforehand. Second, keep a hosted-API path warm permanently. The point of self-hosting is control, and control includes the ability to fail over to something you do not own when your own hardware is down.

Related questions

How much does a self-hosted LLM stack cost to start?

A single-GPU workstation capable of running a quantized 70B model runs in the mid-four to low-five figures depending on the card. Add electricity in the tens of dollars monthly and a meaningful fraction of an engineer's time. Small-model setups on consumer cards cost substantially less.

Can I run a useful model without a GPU?

Yes, for batch work. CPU inference on a modern many-core machine handles 7B-8B class quantized models at single-digit tokens per second — fine for overnight document processing, too slow for interactive chat. Apple Silicon with large unified memory is the better no-discrete-GPU option.

Should I fine-tune or use retrieval?

Start with retrieval. It is cheaper, faster to iterate, and keeps knowledge updatable without retraining. Fine-tune only when you need consistent format, tone, or a task-specific behavior that prompting and retrieval cannot produce reliably after genuine effort.

How do I keep the model current?

Treat model weights as versioned artifacts. Pin a specific version in production, test new releases against your evaluation set in staging, and promote only on measured improvement. Never auto-update a production model — quality regressions are silent and expensive.

What breaks first at scale?

Key-value cache memory under concurrent long-context requests. Aggregate throughput degrades sharply once the engine starts evicting or queueing, and the symptom looks like intermittent latency spikes rather than a clean error. Cap context length and monitor cache utilization as a first-class metric.

FAQ

How much VRAM do I need for a 70B model?

Roughly 140GB at 16-bit precision, about 70GB at 8-bit, and about 35-40GB at 4-bit — before key-value cache. That is why a single 48GB professional card running 4-bit quantization is the standard single-workstation configuration, and why full-precision 70B requires either multiple datacenter GPUs or a large unified-memory system.

Is self-hosting actually cheaper than a hosted API?

Only at sustained high volume. Divide hardware capital cost by your monthly savings over API spend, then add loaded engineering time. If break-even lands past 18-24 months, the economics alone do not justify it — you need a data-control, latency, or air-gap requirement as the real driver.

Do open-weight models match frontier hosted models?

On summarization, extraction, classification, routing, and routine drafting, the gap is small enough that most business workloads will not notice. On the hardest multi-step reasoning and long-horizon agentic tasks, frontier proprietary models still lead. Route by task rather than committing the whole workload to one model.

What inference engine should I pick?

For a single user or a small team wanting the fastest path to running, pick a simple local runner with one-command model pulls. For multi-user production serving, pick an engine built around continuous batching and paged key-value cache with an OpenAI-compatible endpoint. The decision is concurrency, not preference.

Can I use AMD or Apple hardware instead of NVIDIA?

Yes, with caveats. AMD support through ROCm and portable backends like Vulkan works for mainstream engines but typically lags NVIDIA in performance and in day-one support for new features. Apple Silicon runs large models well thanks to unified memory but at lower throughput. NVIDIA remains the path of least resistance.

How do I secure a self-hosted endpoint?

Authenticate every request with API keys or mutual TLS, terminate at a reverse proxy, rate-limit per client, log every request and response for audit, and never expose the inference port to the public internet. Apply the same prompt-injection and least-privilege discipline you would to any hosted model with tool access.

Sources

• Ollama
• Open WebUI on GitHub
• vLLM documentation
• Hugging Face Text Generation Inference
• llama.cpp on GitHub
• NVIDIA CUDA Toolkit documentation
• MLC LLM
• Chroma vector database
• NVIDIA Triton Inference Server

Related on PULSE

• [How do you build a cost dashboard for AI and LLM spend?](/knowledge/ai417)
• [How do you build data pipelines for continuous model training?](/knowledge/ai403)
• [How do you secure an LLM application's infrastructure?](/knowledge/ai363)
• [How do you evaluate LLM output quality at scale?](/knowledge/ai367)
• [The 10 Best LLM Gateways in 2027](/knowledge/ai354)
• [The 10 Best LLM Routing and Load Balancing Tools in 2027](/knowledge/ai412)