How much VRAM do you need to run a 70B parameter model locally in 2027?

A concrete scenario: fitting Llama-class 70B on a desk

Picture a practitioner in 2027 who wants a 70B parameter model running on hardware they own, not rented. They have a workstation with two used 24 GB cards, 128 GB of system RAM, and a fast NVMe drive. Their goal is conversational throughput of at least 10 tokens per second with a usable context window of 8K tokens. The question is not "can a 70B model run at all" — it is "what is the smallest VRAM budget that keeps the model resident and still leaves room for the KV cache."

That framing matters because a 70B model is not a single number. It is a stack of tensors whose footprint depends on the numeric format, the tokenizer, the context length, and the runtime's overhead. A 4-bit quantized 70B model weighs roughly 35–40 GB of weights alone. Add an 8K-token KV cache and framework overhead and you are already brushing against 48 GB. At 32K context, the same model can push past 55 GB before you generate a single token. So the honest answer is a range, not a figure, and the range is set by decisions the user makes before the first prompt.

The scenario also exposes the difference between "fits in VRAM" and "fits in VRAM comfortably." A model that exactly fills available memory will run, but it leaves no headroom for batching, long prompts, or the allocator's fragmentation. Practitioners who plan for 10–15% free VRAM after loading report far fewer out-of-memory crashes than those who pack to the ceiling. That margin is the single most underrated variable in local 70B deployment.

Finally, the scenario forces a choice between buying one large card and buying several smaller ones. A single 48 GB card is simpler, quieter, and avoids multi-GPU communication overhead, but it costs more per gigabyte and limits future expansion. Two 24 GB cards cost less upfront and scale further, but they introduce tensor-parallel overhead, require the model to be sharded, and can bottleneck on PCIe bandwidth if the cards are not on a fast interconnect. The right answer depends on whether the user values simplicity or cost per token.

How the mechanism actually works

To answer the VRAM question precisely, you have to decompose what actually consumes memory when a 70B parameter model runs locally. There are four buckets: weights, KV cache, activations, and runtime overhead. Each scales differently, and each can be tuned independently.

Weights are the dominant term. A 70B model has 70 billion parameters, and each parameter occupies a number of bits determined by its quantization format. At FP16, each parameter takes 2 bytes, so 70B × 2 = 140 GB. At 8-bit, it is 1 byte per parameter, or roughly 70 GB. At 4-bit, it is 0.5 bytes per parameter, or roughly 35 GB, though real 4-bit formats such as NF4 or GPTQ carry small metadata overheads that push the effective figure to 36–40 GB. This is why quantization is the single biggest lever on VRAM requirements.

The KV cache is the second-largest consumer and the one most often underestimated. Every token the model attends to requires a key and a value vector per layer. For a 70B model with 80 layers, a hidden dimension of 8192, and grouped-query attention with 8 key/value heads, the KV cache per token is roughly 80 layers × 2 (key and value) × 8 heads × 128 head dimension × 2 bytes. That works out to about 320 KB per token at FP16, or roughly 2.6 GB for an 8K context. At 32K context it climbs past 10 GB. At 128K context it can exceed 40 GB, which is why long-context 70B inference is a memory problem as much as a compute problem.

Activations and runtime overhead are smaller but non-trivial. Activations depend on batch size and sequence length; for single-stream inference they are typically 1–3 GB. Runtime overhead — CUDA context, framework buffers, allocator fragmentation — adds another 1–2 GB. These are the numbers that turn a "40 GB model" into a "44 GB deployment."

The practical implication is that VRAM planning is a sum, not a single lookup. A 4-bit 70B model at 8K context needs roughly 40 + 2.6 + 2 + 1.5 ≈ 46 GB. That is why 48 GB is the realistic floor for comfortable 4-bit operation and why 24 GB cards must be paired. It also explains why 8-bit 70B is a 80 GB-class proposition and why FP16 70B is effectively a multi-GPU server workload.

Real numbers, ranges, and benchmarks

The table below summarizes the VRAM envelope for a 70B parameter model across common quantization levels and context lengths. These figures assume grouped-query attention with 8 KV heads, a single concurrent request, and a modern runtime such as llama.cpp, vLLM, or TensorRT-LLM. They are planning estimates, not guarantees — actual usage varies by model architecture and runtime.

From these numbers, several concrete hardware configurations emerge. A single 24 GB card cannot hold a 70B model at any usable quantization. Two 24 GB cards (48 GB total) can run 4-bit 70B at 8K context with a small margin, and can stretch to 32K context if the runtime supports KV cache quantization. A single 48 GB card, such as an RTX 6000 Ada-class or a used A6000, runs 4-bit 70B comfortably at 8K and acceptably at 32K. For 8-bit 70B, you need 80 GB — a single A100 80GB, an H100 80GB, or two 48 GB cards. For FP16 70B, you need at least 160 GB, which means two 80 GB cards or four 48 GB cards.

Throughput benchmarks matter as much as capacity. On two 24 GB cards with tensor parallelism over PCIe 4.0, a 4-bit 70B model typically generates 8–15 tokens per second for a single stream. On a single 48 GB card, expect 15–25 tokens per second. On an 80 GB data-center GPU, 25–40 tokens per second is common. These ranges assume the model is fully resident in VRAM; if any layer spills to system RAM, throughput can drop by an order of magnitude because PCIe bandwidth becomes the bottleneck.

Context length is the variable that most often surprises practitioners. Doubling context from 8K to 16K adds roughly 2.6 GB of KV cache, which can be the difference between fitting and not fitting on a 48 GB card. Techniques such as KV cache quantization to 8-bit or 4-bit can cut that growth by half or more, at a small cost in output quality. Sliding-window attention and paged attention also reduce peak KV memory by not materializing the full cache at once. These are the levers to pull when you are 2–4 GB over budget.

Multi-GPU scaling introduces its own arithmetic. With tensor parallelism across two cards, each card holds half the weights and half the KV cache, but each also holds a full copy of the activations and runtime overhead. So two 24 GB cards do not give you a clean 48 GB pool; they give you roughly 44–46 GB of usable model memory after duplicated overhead. That is still enough for 4-bit 70B at 8K, but it is tighter than the raw sum suggests. Pipeline parallelism avoids some duplication but adds latency and complexity.

Trade-offs and alternatives

The central trade-off is between VRAM capacity, quantization level, and output quality. Lower-bit quantization reduces VRAM but can degrade reasoning, instruction-following, and long-context coherence. In practice, 4-bit quantization of a 70B model often outperforms an 8-bit or FP16 version of a much smaller model, which is why 4-bit 70B is the sweet spot for local deployment. But 4-bit is not free: some tasks, particularly those requiring precise arithmetic or strict formatting, show measurable degradation. Practitioners should benchmark on their own workload rather than trusting aggregate scores.

A second trade-off is between buying one large card and several smaller ones. A single 48 GB card is simpler to cool, quieter, and avoids multi-GPU synchronization. Two 24 GB cards cost less per gigabyte and can be upgraded incrementally, but they require a motherboard with enough PCIe lanes, a power supply with enough headroom, and a runtime that supports tensor parallelism. They also suffer from inter-GPU communication overhead, which reduces tokens per second compared to a single card of equal total memory.

A third trade-off is between local and hosted inference. Running a 70B model locally in 2027 costs roughly the price of the hardware plus electricity, but it offers privacy, no per-token fees, and no rate limits. Hosted inference costs nothing upfront but charges per token and sends data to a third party. For occasional use, hosted is cheaper. For sustained daily use, local hardware pays back within months. The break-even point depends on utilization, but it is typically in the range of a few hundred million tokens.

There are also alternatives to full-precision or full-model deployment. Mixture-of-experts models can activate only a fraction of parameters per token, reducing compute but not necessarily VRAM, since all experts must be resident. Speculative decoding uses a small draft model to propose tokens and a large model to verify them, improving throughput without changing VRAM much. Offloading layers to system RAM via mmap lets a model larger than VRAM run, but at a severe throughput penalty. Each of these changes the VRAM equation in a different way, and none of them eliminates the need to plan capacity.

Common pitfalls and how to avoid them

The most common pitfall is planning for weights only and forgetting the KV cache. A practitioner buys a 48 GB card, loads a 4-bit 70B model that occupies 40 GB, and then discovers that an 8K context pushes total usage to 46 GB, leaving almost no headroom. The fix is to budget for KV cache explicitly: decide your target context length first, compute the KV footprint, and subtract it from available VRAM before choosing a quantization level.

The second pitfall is assuming multi-GPU memory pools cleanly. Two 24 GB cards do not behave like one 48 GB card. Each card holds a full copy of activations and runtime overhead, and tensor parallelism adds communication buffers. The usable model memory is typically 90–95% of the raw sum. Practitioners should plan for 44–46 GB usable on a dual-24 GB setup, not 48 GB.

The third pitfall is ignoring PCIe bandwidth. If the model does not fit entirely in VRAM and layers spill to system RAM, throughput collapses. A 70B model running with 10% of layers on CPU can drop from 15 tokens per second to 1–2 tokens per second. The fix is to ensure the model fits entirely in VRAM, or to accept the throughput penalty knowingly. Fast NVMe and high-bandwidth PCIe help, but they do not eliminate the bottleneck.

The fourth pitfall is over-quantizing to fit a smaller card. Running a 70B model at 2-bit or 3-bit quantization to squeeze it onto a 24 GB card often produces output quality worse than a well-quantized 13B or 34B model. The point of running 70B is capability; if quantization destroys that capability, the exercise is self-defeating. Practitioners should compare a 4-bit 70B against a higher-precision smaller model on their actual tasks before committing.

The fifth pitfall is neglecting power and thermals. Two 24 GB cards can draw 600–700 watts under load, and a single 48 GB workstation card can draw 300–450 watts. A consumer power supply and case may not handle that. Practitioners should budget for a 1000W or larger PSU, adequate case airflow, and possibly a dedicated circuit. Thermal throttling can silently reduce throughput by 20–30%, which is easy to mistake for a software problem.

The sixth pitfall is assuming software support is uniform. Not every runtime supports every quantization format, every multi-GPU topology, or every KV cache optimization. A configuration that works in llama.cpp may not work in vLLM, and vice versa. Practitioners should verify that their chosen runtime supports their target quantization, context length, and GPU count before buying hardware. Community benchmarks and release notes are the best source for this, but they should be treated as directional rather than definitive.

Related questions

Can a single 24 GB GPU run a 70B model locally?

Not at usable quality. A 4-bit 70B model needs roughly 36–40 GB for weights alone, before KV cache and overhead. A 24 GB card can only run heavily quantized 70B at very short context, with quality loss that often makes a smaller model the better choice.

Is 4-bit quantization good enough for a 70B model?

For most conversational and reasoning tasks, yes. A 4-bit 70B model typically outperforms an 8-bit or FP16 model of much smaller size. However, tasks requiring precise arithmetic, strict formatting, or long-context coherence may show degradation, so benchmark on your own workload.

How much system RAM do you need alongside VRAM?

Plan for at least 64 GB of system RAM, and 128 GB is safer. System RAM holds the model file during loading, the operating system, and any offloaded layers. If you offload, more RAM helps, but throughput still drops because PCIe bandwidth becomes the bottleneck.

Does context length affect VRAM more than quantization?

Both matter, but they scale differently. Quantization changes the weight footprint, which is the largest single term. Context length changes the KV cache, which grows linearly and can add 10 GB or more at 32K. A 4-bit model at 32K can need more VRAM than an 8-bit model at 4K.

What is the cheapest hardware path to 4-bit 70B in 2027?

Two used 24 GB cards on a PCIe 4.0 motherboard is typically the cheapest path, assuming the rest of the system can support them. A single 48 GB card costs more but is simpler. Cloud rental is cheaper for occasional use but not for sustained daily inference.

FAQ

How much VRAM do you need to run a 70B parameter model locally in 2027? Plan for roughly 40–48 GB at 4-bit quantization, 70–80 GB at 8-bit, and 140 GB or more at FP16. These figures include weights, KV cache at 8K context, activations, and runtime overhead. Add 10–15% headroom for safety.

Why is 4-bit the most common choice for local 70B deployment? Because it cuts the weight footprint from 140 GB to roughly 36–40 GB, which brings 70B within reach of 48 GB workstations and dual-24 GB setups. The quality loss is usually small relative to the memory savings, making it the best capacity-per-bit trade-off.

Can you run a 70B model with less than 40 GB of VRAM? Only with aggressive quantization or offloading. A 3-bit or 2-bit 70B model can fit in 24–30 GB, but quality often drops below a well-quantized smaller model. Offloading layers to system RAM works but reduces throughput to 1–2 tokens per second.

Does the KV cache really matter for a 70B model? Yes. At 8K context the KV cache is roughly 2.6 GB, at 32K it is about 10 GB, and at 128K it can exceed 40 GB. For long-context workloads, the KV cache can rival the weights as the dominant VRAM consumer.

What is the difference between tensor parallelism and pipeline parallelism for VRAM? Tensor parallelism splits each layer across GPUs, so each card holds a fraction of the weights and KV cache, but duplicates activations and overhead. Pipeline parallelism splits layers into stages, reducing duplication but adding latency. Tensor parallelism is more common for 70B inference.

Will 70B models still need this much VRAM in 2027? Architecture improvements such as grouped-query attention, KV cache quantization, and mixture-of-experts reduce memory per token, but the fundamental relationship between parameter count and weight size remains. A 70B model at 4-bit will still need roughly 36–40 GB for weights in 2027.

Sources

• https://huggingface.co/docs/transformers/main/en/quantization
• https://github.com/ggml-org/llama.cpp
• https://docs.vllm.ai/en/latest/
• https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
• https://arxiv.org/abs/2305.13245
• https://arxiv.org/abs/2306.00978
• https://www.nvidia.com/en-us/data-center/a100/
• https://www.nvidia.com/en-us/design-visualization/rtx-6000/

Related on PULSE

• How much VRAM do you need to run a 13B parameter model locally?
• How much VRAM do you need to run a 34B parameter model locally?
• Quantization formats compared: GPTQ, AWQ, GGUF, and NF4
• Multi-GPU inference: tensor vs pipeline parallelism
• KV cache optimization techniques for long-context inference
• Local vs hosted inference: cost and privacy trade-offs