What is the difference between vLLM, TGI, and Triton for LLM inference?

What it is and why it matters

The most common mistake teams make when comparing these three is assuming they occupy the same layer of the stack. They do not, and that single misunderstanding drives most bad architecture decisions in self-hosted inference.

vLLM is an inference *engine*. It was developed at UC Berkeley and its defining contribution is PagedAttention — a KV-cache memory manager that borrows the virtual-memory paging idea from operating systems. Instead of allocating one contiguous slab of GPU memory per sequence sized to the maximum possible output length, vLLM allocates fixed-size blocks on demand and maintains a block table per sequence. The practical consequence is that memory that would otherwise sit reserved-but-unused gets recycled into additional concurrent requests. vLLM ships a server with OpenAI-compatible /v1/chat/completions and /v1/completions endpoints, so most application code written against a hosted API points at it with a base-URL change.

TGI (Text Generation Inference) is also an inference engine plus server, built and maintained by Hugging Face. Its design center is different: TGI optimizes for the path from "a model exists on the Hugging Face Hub" to "that model is serving traffic." It handles tokenizer resolution, weight download, sharding across GPUs, and quantization format detection largely from the model repo's own config. TGI implements continuous batching, FlashAttention, tensor parallelism, and support for bitsandbytes, GPTQ, and AWQ quantized weights. Recent versions have adopted paged KV-cache management as well, which has narrowed the raw-throughput gap that existed in 2023–2024.

Triton Inference Server is a *serving platform*, not an LLM engine. It predates the current LLM wave by years and was designed to serve any model: computer vision, recommendation, tabular, ASR, and now LLMs. Triton's job is protocol handling (HTTP and gRPC), request scheduling, dynamic batching, model repository management, versioning, concurrent execution of multiple models on one GPU, and Prometheus metrics. The actual LLM math happens in a *backend* — most commonly TensorRT-LLM, but Triton also has a Python backend, and vLLM itself can be run as a Triton backend. So "vLLM vs Triton" is frequently a category error; the real comparison is "vLLM standalone vs Triton + TensorRT-LLM," or even "vLLM standalone vs Triton wrapping vLLM."

Why this matters to revenue and not just to engineering: inference cost is close to pure marginal cost of goods sold for any AI product. If your GPU fleet serves 40% more concurrent users at the same hardware spend, gross margin on every AI feature moves accordingly. A team that picks the wrong layer — building a multi-model routing system on top of a single-model engine, or standing up Triton's full model-repository machinery to serve exactly one 7B model — pays for it in either wasted GPU-hours or wasted engineering weeks. Both show up on the same P&L.

The step-by-step process

Here is the evaluation sequence that actually produces a defensible decision, rather than a benchmark argument on the internet.

Step 1 — Write down the workload shape, in numbers. Before touching any of the three, record: number of distinct models you must serve; average and p95 input token length; average and p95 output token length; requests per second at peak; whether requests are streaming or batch; and your latency SLO expressed as either time-to-first-token (TTFT) or end-to-end. A RAG chat product with 4,000-token prompts and 200-token answers is a fundamentally different serving problem from a summarization batch job with 500-token prompts and 1,500-token answers. The first is prefill-bound and benefits enormously from prefix caching; the second is decode-bound and benefits from large batch sizes.

Step 2 — Eliminate on hard constraints. If you must serve models from multiple frameworks in one process (say a sentence-transformer embedder, an ONNX reranker, and an LLM), that constraint alone points to Triton. If you must run on non-NVIDIA hardware, TensorRT-LLM is out, which effectively removes the highest-performance Triton configuration. If your team's entire model workflow lives in Hugging Face transformers and you fine-tune with PEFT and push to the Hub, TGI removes an integration step.

Step 3 — Build a benchmark harness before installing anything. Use a load generator that replays *your* prompt distribution, not a synthetic fixed-length one. Fixed-length benchmarks systematically flatter engines with simpler memory managers, because variable-length output is precisely where paged KV cache earns its keep. Record tokens/second aggregate throughput, TTFT at p50/p95/p99, inter-token latency, and GPU memory high-water mark.

Step 4 — Run each candidate at three concurrency levels. Low (1–4 concurrent), medium (16–32), and saturation (whatever level pushes p95 TTFT past your SLO). Engines rank differently at different concurrency. At concurrency 1, all three are within noise of each other, because you are bound by raw kernel speed and memory bandwidth, not by scheduling. Differences appear at medium-to-high concurrency where batching strategy and KV-cache efficiency dominate.

Step 5 — Measure the operational cost, not just the throughput. Time how long it takes an engineer unfamiliar with the tool to get a model serving from scratch, and how long a model swap takes. For vLLM and TGI this is typically a single docker run plus a model identifier. For Triton with TensorRT-LLM it involves compiling an engine artifact for a specific GPU architecture, batch size range, and sequence length range — and that artifact must be rebuilt when any of those change or when you move from A100 to H100.

Step 6 — Decide, then re-measure quarterly. All three projects ship frequently. A throughput conclusion drawn from a release six months old should be treated as expired.

Costs, timelines, and typical ranges

Every one of these three is open source and free of license cost. The real spend is GPU time and engineering time, and the ranges below are the ones worth planning against.

Engineering time to first served request. vLLM and TGI both land in the same bucket: a containerized server pointed at a model identifier, running in well under an hour for someone who has done it once, and typically inside a working day for someone who has not — most of that day spent on CUDA driver and container-toolkit setup rather than the engine itself. Triton with TensorRT-LLM is a different order of magnitude. You are building an engine artifact, laying out a model repository directory with config.pbtxt files, and often assembling an ensemble that chains a tokenizer step, the TensorRT-LLM step, and a detokenizer step. Budget one to two weeks for a first production-grade Triton LLM deployment, and expect a meaningful chunk of that to be spent on config semantics rather than on performance.

Model swap and iteration cost. This is the cost line teams underestimate. Swapping models in vLLM or TGI means changing a model path and restarting — minutes. Swapping models in a TensorRT-LLM setup means recompiling an engine for the new architecture, which is a build step measured in tens of minutes, plus revalidation. If your product roadmap involves trying a new base model every few weeks, that difference compounds into real calendar time.

GPU memory planning. The dominant variable is KV cache, not weights. Weights are predictable: roughly 2 bytes per parameter at FP16/BF16, so a 7B model is about 14 GB and a 70B model about 140 GB before any quantization. KV cache is what scales with concurrency and context length, and it is where the three systems diverge. With naive contiguous allocation, you must reserve cache for the *maximum* sequence length for every in-flight request, so a request that generates 50 tokens holds memory sized for the 4,096-token worst case. Paged allocation reserves in small blocks and releases them as sequences finish. The practical outcome is that on identical hardware and identical weights, a paged system supports substantially more concurrent sequences before it starts rejecting or queueing. This is the single largest lever on cost-per-token in self-hosted inference.

Quantization as a cost lever. All three support quantized weights, with somewhat different format coverage. TGI has strong coverage of bitsandbytes, GPTQ, and AWQ. vLLM supports GPTQ, AWQ, and FP8 on hardware that provides it. TensorRT-LLM supports FP8, INT8, and INT4 with NVIDIA's own calibration tooling and, on Hopper-class and newer GPUs, gets the most out of native FP8 tensor cores. Moving a 70B model from BF16 to a 4-bit format takes weights from roughly 140 GB to roughly 35–40 GB, which is the difference between needing a multi-GPU node and fitting on a single large-memory card. That is a direct hardware line-item change, and it is usually the highest-leverage cost decision available — larger than the engine choice itself.

Throughput expectations. Be careful with any specific tokens/second figure you read, including in vendor blog posts, because the number is meaningless without the model, GPU, quantization, input/output length distribution, and concurrency level attached. What holds up across published comparisons is the *shape* of the result: at single-request concurrency the three are close; as concurrency rises, engines with paged KV cache and continuous batching pull ahead of those without; and TensorRT-LLM on NVIDIA hardware with a compiled engine and FP8 tends to occupy the top of the range at the cost of the build step. Generate your own absolute numbers on your own hardware — that is the only figure you can defend in a capacity plan.

Ongoing operational overhead. Triton gives you Prometheus metrics, model versioning, and concurrent model execution out of the box; if you need those, building them around vLLM or TGI is real work. Conversely, if you do not need them, Triton's configuration surface is overhead you pay for nothing.

Where teams get it wrong

Treating Triton as a competitor to vLLM. As covered above, Triton is a serving platform and vLLM is an engine. Teams write comparison docs pitting them head to head, pick one, and then discover six months later that the actual answer was "Triton in front, vLLM as the backend." Frame the decision as two separate questions — which engine does the token math, and which server handles protocol, scheduling, and multi-model concerns — and the architecture usually resolves itself.

Benchmarking with fixed-length synthetic prompts. This is the single most common measurement error. A benchmark where every request has exactly 512 input tokens and generates exactly 128 output tokens hides the entire advantage of paged memory management, because with uniform lengths there is little fragmentation to eliminate. Real traffic has a long tail of both prompt and completion lengths. Replay a sample of production traffic, or at minimum sample lengths from a realistic distribution.

Optimizing aggregate throughput when the product is latency-sensitive. Throughput and latency trade against each other through batch size. A larger batch raises tokens/second in aggregate while raising time-to-first-token for every request in it. If you are serving an interactive chat interface, users perceive TTFT and inter-token latency, not fleet throughput. Pick the metric that matches the product before you tune, and set an explicit p95 TTFT budget.

Ignoring prefix caching in RAG workloads. In retrieval-augmented generation, a large system prompt and often a large retrieved-context block repeat across requests. Engines that cache the KV state for a shared prefix skip recomputing attention over those tokens entirely. When your prompts are 4,000 tokens and 3,000 of them are a stable prefix, this is not a marginal optimization — it changes the prefill cost profile of the whole service. Confirm the feature is actually enabled; it is not always on by default.

Compiling a TensorRT-LLM engine and forgetting it is hardware-pinned. The compiled artifact targets a specific GPU architecture and a specific range of batch sizes and sequence lengths. Move from A100 to H100, or raise your max sequence length, and you rebuild. Teams that skip documenting the build command and its parameters discover this during an incident, which is the worst possible time.

Running one giant instance instead of several smaller ones. A single instance holding one model is simple, but it makes rolling upgrades and blast-radius control harder. Multiple replicas behind a load balancer let you drain one at a time. This matters more than engine choice for actual availability.

Assuming an OpenAI-compatible endpoint means full compatibility. vLLM and TGI both expose OpenAI-shaped APIs, and both cover the common path well. The edges — specific tool-calling formats, logprobs, seed behavior, structured-output modes — vary by version. Test the exact fields your application sends before you commit.

Not accounting for the tokenizer. In a Triton ensemble, tokenization and detokenization are explicit pipeline stages you configure. In vLLM and TGI they are handled internally. Teams migrating from one to the other sometimes introduce subtle token-boundary differences that change model output. Diff a sample of generations across engines before cutting traffic over.

Decision framework: when to choose what

Reduce the decision to a small number of binary questions, answered in order. The first question that produces a "yes" usually determines the answer.

**Do you need to serve more than one *kind* of model in a single deployment?** If your service must run an LLM alongside an embedding model, a reranker, a classifier, or a vision model — and you want them sharing GPUs with a common scheduler, metrics, and versioning story — that is Triton's purpose and none of the alternatives cover it as cleanly. Ensembles let you express the whole chain as one server-side pipeline, so the client makes one call instead of orchestrating several.

Is your entire model lifecycle already on Hugging Face? If your team fine-tunes with transformers/PEFT, versions weights on the Hub, and wants deployment to be a near-zero-config step from that repo, TGI removes the most friction. The value here is not peak tokens/second; it is that the tokenizer, config, and quantization format come along automatically and the number of places a mismatch can occur drops.

Are you serving one or a few LLMs at high concurrency, on NVIDIA GPUs, where cost-per-token is the dominant concern? This is vLLM's core case. You get paged KV cache, continuous batching, prefix caching, and an OpenAI-compatible API without a compilation step, and swapping models is a restart.

Do you need the last increment of performance on NVIDIA hardware and can you afford a build step? TensorRT-LLM, typically fronted by Triton, is the answer — with the standing cost that engine artifacts are pinned to hardware and shape parameters and must be rebuilt when either changes. Take this path when you are running at a scale where a further percentage of throughput is worth a permanent build-and-validate pipeline.

Do you need autoscaling, multi-node distribution, or canary/A-B routing? That is an orchestration layer above the engine — Ray Serve, a Kubernetes-native serving stack, or your own routing tier — and it composes with whichever engine you chose. Do not try to solve it inside the engine.

A pattern that works well in practice: vLLM serving the LLM itself, Triton serving the surrounding models, and a routing layer in front. You get vLLM's memory efficiency on the expensive part of the workload and Triton's multi-model machinery where it is genuinely needed, without forcing either tool into a role it was not designed for.

Related questions

Can vLLM run inside Triton?

Yes. Triton supports vLLM as a backend, so you can keep Triton's protocol handling, metrics, model repository, and versioning while vLLM does the token generation. This is often the right answer when a team wants Triton's operational surface without TensorRT-LLM's compilation step.

Does TGI still lag vLLM on throughput?

The large gap reported in 2023–2024 has narrowed considerably as TGI adopted paged KV-cache management and continued optimizing its batching. Any current ranking depends on model, hardware, and traffic shape, so benchmark both on your own workload rather than citing an older comparison.

Which of the three works on AMD or Apple hardware?

TensorRT-LLM is NVIDIA-only. vLLM and TGI have varying degrees of non-NVIDIA support that changes release to release, so verify against current documentation. For Apple Silicon and CPU-first local inference, llama.cpp remains the practical choice rather than any of these three.

Do all three expose an OpenAI-compatible API?

vLLM and TGI both ship OpenAI-shaped chat and completion endpoints directly. Triton exposes its own HTTP and gRPC protocols; getting an OpenAI-compatible surface means either a translation layer in front or using a backend that provides one. Verify field-level behavior before switching.

How much does quantization change the engine decision?

Less than teams expect. Quantization is usually the larger cost lever — it can move a model from multi-GPU to single-GPU — but all three support common quantized formats. Choose the engine on serving architecture, then apply quantization to whichever you picked.

FAQ

What is PagedAttention and why does it matter?

PagedAttention is vLLM's KV-cache memory manager, modeled on operating-system virtual memory. Rather than reserving one contiguous block per sequence sized for the worst-case output length, it allocates small fixed-size blocks on demand and tracks them in a per-sequence block table. Memory freed by a finished sequence is immediately reusable, so more requests fit concurrently on the same GPU. Since KV cache — not weights — is what limits concurrency at long context lengths, this is the mechanism behind most of vLLM's throughput advantage over naive allocation.

Can I use Triton without TensorRT-LLM?

Yes. Triton is backend-agnostic and supports PyTorch, ONNX Runtime, TensorFlow, a Python backend, and vLLM among others. TensorRT-LLM is the highest-performance option for LLMs on NVIDIA GPUs but it carries a compilation step. Running vLLM as a Triton backend is a common middle path: Triton's scheduling, metrics, and model management with vLLM's memory efficiency and no engine build.

Is the difference between these three mainly about speed?

No — and framing it that way causes most of the bad decisions in this space. The primary difference is architectural scope. Triton is a general serving platform for many model types; vLLM and TGI are LLM-specific engines with their own servers. Speed differences between the LLM engines are real but narrow over time as all three ship optimizations, whereas the architectural difference is structural and does not converge.

How should I benchmark these fairly?

Use your own model on your own GPU with a prompt-length and output-length distribution sampled from real traffic, not fixed-length synthetic requests. Test at least three concurrency levels including saturation. Record aggregate throughput, TTFT at p50/p95/p99, inter-token latency, and peak GPU memory. Run each engine long enough to reach steady state, and re-run after any version upgrade — release-to-release changes in this space are large enough to invalidate old results.

What is continuous batching and do all three do it?

Continuous batching (also called in-flight batching) lets the scheduler insert new requests into a running batch as soon as any sequence finishes, instead of waiting for the entire batch to complete. Without it, one long generation stalls every short request batched with it. vLLM, TGI, and TensorRT-LLM behind Triton all implement some form of it; the differences are in scheduling policy and how they interact with preemption and memory pressure.

Does the engine choice actually affect revenue?

Indirectly but materially. Inference is the marginal cost of every AI feature you ship, so concurrency per GPU translates straight into gross margin. It also affects latency, which affects completion and retention on interactive products. The larger levers are usually model size and quantization, but at scale the serving stack is a real line item — worth measuring properly rather than deciding by reputation.

Sources

• vLLM documentation
• vLLM GitHub repository
• Efficient Memory Management for Large Language Model Serving with PagedAttention (arXiv)
• Hugging Face Text Generation Inference documentation
• Text Generation Inference GitHub repository
• NVIDIA Triton Inference Server documentation
• Triton Inference Server GitHub repository
• TensorRT-LLM GitHub repository
• TensorRT-LLM documentation
• Ray Serve documentation

Related on PULSE

• [What is the difference between batch and real-time inference infrastructure?](/knowledge/ai409)
• [How do you choose between cloud GPUs and on-prem for AI workloads?](/knowledge/ai375)
• [What causes high latency in LLM inference and how do you fix it?](/knowledge/ai389)
• [How do you load-test an LLM inference service?](/knowledge/ai425)
• [How do you scale LLM inference to handle thousands of concurrent users?](/knowledge/ai347)