How do you architect a RAG pipeline for low latency?

The outcome you should expect

The number that matters to a user is time-to-first-token, not total wall-clock. A well-built pipeline lands retrieval in the 5–25 ms band and puts the first visible character on screen in roughly 400–900 ms, with the remainder of the answer streaming behind it. That is the target to hold yourself to. Teams who instrument for the first time are usually surprised by where the time actually sits: retrieval is rarely the villain. In a typical unoptimized stack, a remote embedding API call eats 80–250 ms, the vector search itself takes 10–40 ms, an optional cross-encoder rerank adds 50–300 ms, prompt assembly and token counting take 5–20 ms, and then the LLM's own prefill plus first-token generation accounts for 300–2000 ms depending on context length and model size. Retrieval is maybe 5% of the budget. The embedding hop and the rerank hop are the two places where naive designs quietly donate half a second.

So the outcome you should expect from a serious latency effort is not "the vector database got faster." It is: the embedding call moved in-process, the rerank became conditional, the context got shorter, and the response began streaming immediately. Those four changes routinely take a 2.5-second perceived wait down to under 700 ms without touching the retrieval engine at all.

Set explicit service-level objectives before you tune anything, because "fast" is not a spec. A workable pair for an interactive assistant: p50 time-to-first-token under 600 ms, p99 under 1.8 s. For an internal analytics copilot where users tolerate a beat of thinking, p99 of 3 s is fine and you can afford reranking and larger context. For a voice or telephony agent, the ceiling is brutal — humans read silence over roughly 300 ms as a dropped call, so you need sub-300 ms first-audio, which forces filler-phrase playback while retrieval runs underneath. Different SLOs produce genuinely different architectures; do not build one pipeline and hope it serves all three.

The commercial framing matters too, because latency work competes for engineering time against features. Interactive latency is a conversion variable, not a vanity metric. Support deflection rates, in-product search engagement, and demo-to-close motion all move when an assistant answers in under a second versus three seconds, and that is where a latency budget earns its keep in revenue terms rather than in benchmark screenshots. Frame the work that way when you ask for the sprint.

What drives that outcome

Five levers do almost all the work, roughly in order of payoff per hour of engineering.

Cache at three layers. An exact-match response cache on the normalized query string is trivially cheap and, in production RAG over a finite corpus, hit rates of 15–40% are common because real user questions cluster hard. A semantic cache — embed the query, look for a prior query above ~0.95 cosine similarity, return the stored answer — extends that but is riskier: set the threshold too loose and you serve a confidently wrong answer to a subtly different question. Start at 0.97 and loosen only with eval data. The third and most underrated layer is an embedding cache: store query-string → vector so repeat and near-repeat queries skip the encoder entirely. All three can live in Redis with a TTL keyed to how often your corpus changes.

Embed locally, not over the network. A hosted embedding endpoint costs you a full network round trip on the critical path, and that round trip has a fat tail — p99s of 500 ms+ are ordinary when the provider is having a moment. A small sentence-transformer running in-process on CPU encodes a short query in 3–15 ms; on a modest GPU, under 2 ms. The quality gap against a large hosted embedder is real but far smaller than most teams assume for retrieval-over-your-own-docs, and you can hedge by using the big model for offline document embedding and a distilled model for online query embedding only if the two share a vector space. If they do not, you must use the same model on both sides — mixing embedding spaces silently destroys recall, and it is one of the most common self-inflicted wounds in this whole space.

Keep the index in memory and pick HNSW. HNSW gives sub-10 ms search over a million 768-dimensional vectors on a single node and degrades gracefully under concurrency. IVF variants need more tuning and shine mainly at billion scale. DiskANN and other SSD-backed indexes trade roughly 10–30 ms of extra latency for an enormous cost reduction, which is the right trade for large archival corpora and the wrong one for a chat surface. Quantization is the cheapest win available: moving float32 vectors to int8 cuts memory around 4× with single-digit-percent recall loss, and binary quantization with a small float rescoring pass over the top candidates gets you most of the way back on quality at a fraction of the footprint.

Retrieve less. Every additional chunk you stuff into the prompt lengthens LLM prefill, and prefill scales with input tokens. Going from top-20 at 1000 tokens per chunk to top-5 at 400 tokens cuts input from 20k to 2k and can halve time-to-first-token outright. Chunk size is a latency parameter, not just a quality parameter — 300–600 token chunks with 10–15% overlap is a sane default that keeps context tight.

Make reranking conditional. A cross-encoder reranker genuinely improves precision, and it genuinely costs 50–300 ms. Run it only when the retrieval scores are ambiguous — for instance when the gap between the top result and the fifth result is small, signalling the vector search did not find a clear winner. On confident retrievals, skip it. In practice this fires on a minority of queries and buys you most of the quality for a fraction of the average latency.

Benchmarks and realistic ranges

Treat every published vector-database benchmark as a claim about someone else's hardware, someone else's dimensionality, and someone else's recall target. The only number that matters is the one you measure on your corpus. That said, some ranges are stable enough to plan against.

In-memory HNSW over a million vectors at 768 dimensions, with a single-digit top-k and recall around 0.95, sits in the 2–10 ms range on a decent server core, and p99 stays under about 30 ms at moderate concurrency. Push to 10 million vectors and expect roughly 10–40 ms unless you shard. Move the index to disk and add 10–30 ms. Add a network hop to a managed service in the same cloud region and add 1–5 ms; cross-region, add 30–80 ms, which is why co-locating the index with the application is non-negotiable for interactive workloads. Memory planning is simple arithmetic: a million float32 vectors at 768 dimensions is about 3 GB of raw vectors, plus HNSW graph overhead that typically runs 20–50% on top depending on your M parameter. Int8 quantization takes that 3 GB to roughly 800 MB.

The tuning knobs that actually move the needle on HNSW are M (graph connectivity, typically 16–48) and efSearch (candidate list size at query time, typically 40–200). efSearch is the live latency-versus-recall dial — it is safe to change at runtime, and the honest way to use it is to build a small recall-versus-latency curve on your own data and pick the knee rather than accepting a default. Raising efSearch from 64 to 256 might buy you two points of recall for triple the search time; whether that trade is worth it depends entirely on whether those two points change answer quality, which only an eval set can tell you.

On the generation side, the dominant variable is input length. Prefill is roughly linear in input tokens, so a 16k-token context does not cost twice a 8k-token context — it costs about twice, plus whatever attention overhead your serving stack carries. Streaming changes the user's perception fundamentally: with streaming, only prefill plus one token is on the critical path, and everything after is amortized against reading speed. Without streaming, the user waits for the full generation, which for a 500-token answer is often 3–8 seconds. Enabling streaming is the single highest-leverage change in most RAG stacks and it is a client-side concern as much as a server one.

Measure percentiles, never averages. A pipeline with a 200 ms mean and a 4 s p99 feels broken, because the p99 lands on your most engaged users — the ones asking the long, unusual questions that miss every cache. Instrument each stage separately with spans, and log the stage breakdown on any request that exceeds your SLO so you have the evidence when someone claims "the vector DB is slow." Nine times out of ten it is the embedding provider, a cold container, or a context that ballooned to 30k tokens because a chunking bug produced duplicates.

Load-test with realistic concurrency and realistic query diversity. Replaying the same query a thousand times measures your cache, not your pipeline. Sample real production queries, or generate a diverse set from your corpus, and hold concurrency at your expected peak for long enough that connection pools, JIT warmup, and garbage collection all show themselves.

Risks, edge cases, and failure modes

Cold starts are the most common gap between a benchmark and reality. A serverless function that loads an embedding model on first invocation pays 2–10 seconds on that request. If your traffic is bursty, a meaningful share of users hit cold containers. The fixes are unglamorous: provisioned concurrency, a warm pool, a keepalive ping, or simply running the service on a long-lived container. Index warmup has the same shape — an HNSW graph loaded from disk has cold pages, and the first few hundred queries run slower until the working set is resident.

Semantic caching is where correctness risk concentrates. Two queries can be 0.96 cosine-similar and have opposite correct answers — "can I cancel my plan" and "can I cancel my plan after renewal" are near-neighbors semantically and different questions operationally. Any cached answer that encodes a user-specific or time-sensitive fact must be keyed by tenant and given a short TTL, or you will eventually serve one customer's data to another. Never cache across tenant boundaries without the tenant ID in the key. This is the single most likely way a latency optimization turns into a security incident.

Staleness is the tax you pay for aggressive caching. If your corpus updates hourly, a 24-hour cache TTL means users get yesterday's answer with today's confidence. Wire cache invalidation to your ingestion pipeline — when a document is re-indexed, evict cache entries whose retrieved chunks came from it. That requires storing the source-document IDs alongside each cached answer, which is a small amount of bookkeeping that saves a large amount of embarrassment.

Concurrency behavior is often nonlinear and surprising. Many vector engines are fine at 10 concurrent queries and fall off a cliff at 200 because of thread-pool saturation or lock contention on the index during writes. If you index continuously while serving, you are contending — separate read and write replicas, or batch your ingestion into windows. Under sustained load, tail latency degrades long before mean latency does, so a mean-based autoscaling policy will scale too late.

Quantization failure modes are quiet. Int8 usually costs a few points of recall, which shows up not as an error but as answers that are subtly less grounded — the right document sat at rank 12 instead of rank 3 and got trimmed. You will not notice without a retrieval eval set that measures recall@k on known question-document pairs. Build that eval set before you quantize, not after.

Two more traps worth naming. First, mixing embedding model versions across a corpus — if you re-embed half your documents with a new model and leave the rest, the vector space is incoherent and retrieval quality collapses in a way that looks random. Re-embed atomically into a new collection and swap. Second, unbounded context growth: a chunking bug or a duplicate-ingestion bug can silently double your prompt size, and since the system still returns correct answers, nobody notices until the latency graph drifts upward over a month. Alert on p95 input-token count, not just latency.

Finally, the graceful-degradation path. When the reranker times out, return the unreranked results rather than failing. When the embedding service is down, fall back to lexical BM25 search — worse, but answering. When the vector store is unreachable, answer from the model's parametric knowledge with an explicit caveat, or return a clean "I could not retrieve sources" rather than a hung request. Every stage on the critical path needs a timeout and a fallback, because a pipeline that is fast at p50 and hangs at p99.9 is worse than one that is uniformly mediocre.

A practical rollout plan

Do this in order. Each step is independently shippable and each one tells you whether the next is worth doing.

Week one: instrument. Add per-stage timing spans — query normalization, cache lookup, embedding, vector search, rerank, prompt assembly, LLM prefill, first token, full completion. Log input token count. Ship this before changing anything, because otherwise you will optimize the wrong stage. Build the p50/p95/p99 dashboard and let it run for a week against real traffic so you know your actual distribution and your actual cache-hit ceiling.

Week two: streaming and the exact cache. Turn on token streaming end to end, including through any proxy or gateway that might be buffering — a surprising number of latency problems are a reverse proxy holding the response until it completes. Add the exact-match response cache with a conservative TTL. These two changes are low-risk and typically deliver the largest perceived improvement of the entire project.

Week three: move embedding in-process. Swap the hosted query-embedding call for a local model, verifying that document and query embeddings share a space. Measure recall on your eval set before and after; if recall drops more than a couple of points, keep the hosted model for documents and reconsider. Add the embedding cache at the same time.

Week four: tune retrieval. Sweep efSearch and top-k against your eval set and pick the knee of the recall-latency curve. Shrink chunk size if your chunks are large. Make reranking conditional on score ambiguity. Then, and only then, consider quantization — with the eval set in place to catch recall regressions.

Ongoing: keep the SLO honest. Alert on p99 time-to-first-token and on p95 input tokens. Re-run the recall eval whenever you change the embedding model, the chunker, or the index parameters. Review cache-hit rate monthly; a falling hit rate usually means your user base or corpus shifted and your TTLs need revisiting.

The adjacent workflows benefit from the same discipline. Agentic pipelines that call retrieval two or three times per turn multiply every millisecond you failed to remove, so a 200 ms retrieval step that felt acceptable in single-shot RAG becomes 600 ms of dead air in a multi-hop agent. Batch document ingestion has the inverse profile — throughput matters, latency does not, so use the large embedding model and large batches there and save the small model for the online path. Internal search, recommendation surfaces, and duplicate detection all sit on the same vector infrastructure and inherit the same cache and index tuning, which means the work you do here usually pays off across three or four surfaces rather than one.

Related questions

Does the vector database choice actually matter for latency?

Less than teams expect. Any competent in-memory HNSW implementation lands in the same single-digit-millisecond band. Choice matters far more for operational fit — hybrid search, filtering, multi-tenancy, replication, and cost — than for raw query speed on the critical path.

Should I use a reranker at all?

Use one when precision matters and you can afford 50–300 ms, but make it conditional rather than unconditional. Run it only when top-k scores are tightly clustered, which signals the vector search found no clear winner. Most queries skip it entirely.

How much does chunk size affect latency?

Substantially, through prompt length rather than search time. Halving chunk size roughly halves input tokens for the same top-k, which cuts LLM prefill proportionally. Chunks of 300–600 tokens with modest overlap balance retrieval quality against context cost.

What is a reasonable time-to-first-token target?

Under 600 ms at p50 and under 1.8 s at p99 for an interactive assistant. Voice agents need sub-300 ms first audio, usually achieved with filler phrases masking retrieval. Batch or analytical use cases can comfortably accept several seconds.

Can hybrid search be added without a latency penalty?

Usually yes if the engine runs lexical and vector search in parallel and fuses the results, which costs the slower of the two rather than the sum. Sequential implementations do add latency, so verify how your engine executes it.

FAQ

What is the single biggest latency win in a RAG pipeline?

Enabling token streaming, followed by moving query embedding out of a remote API call and into the application process. Streaming changes perceived latency from full-generation time to prefill time, often a 3–5× improvement in how fast the system feels. The in-process embedder removes a network round trip with a heavy tail. Neither requires changing your vector store.

How do I know whether retrieval or generation is my bottleneck?

Instrument each stage with separate timing spans and look at the breakdown on slow requests specifically, not on the average. In most stacks retrieval is under 10% of end-to-end time. If your dashboard shows retrieval dominating, look for a cross-region hop, a cold index, or write contention from concurrent ingestion rather than assuming the engine is slow.

Is quantization safe to turn on?

Int8 quantization is generally safe and cuts memory roughly 4× with single-digit-percent recall loss. Binary quantization is more aggressive and should be paired with a float rescoring pass over the top candidates. In both cases, build a retrieval eval set measuring recall@k on known question-document pairs first, because quantization damage shows up as subtly worse grounding, not as errors.

How should I handle multi-tenant caching?

Include the tenant identifier in every cache key, without exception, and never let a semantic cache match across tenant boundaries. Give tenant-scoped entries short TTLs and wire eviction to your ingestion pipeline so re-indexed documents invalidate answers derived from them. The performance gain from cross-tenant sharing is never worth the data-exposure risk.

Does GPU acceleration help retrieval latency?

For most corpus sizes, no — a CPU HNSW index already answers in single-digit milliseconds, and the GPU mainly helps at very large scale or with brute-force search. GPUs help far more on the embedding and generation stages. Spend GPU budget on serving the model and the encoder before spending it on the index.

What is the right way to handle a slow or failing retrieval stage?

Set an explicit timeout on every critical-path stage and define a fallback for each. Reranker timeout returns unreranked results; embedding failure falls back to lexical search; vector store unavailability returns an honest "sources unavailable" response. Uniform mediocrity beats a fast median with a hanging tail.

Sources

• Redis Vector Search documentation
• Qdrant benchmarks
• Weaviate performance and benchmarks
• pgvector repository and index documentation
• Milvus index and performance documentation
• Chroma documentation
• hnswlib — reference HNSW implementation
• Faiss documentation and index guidelines
• Sentence-Transformers documentation
• Vespa vector search documentation

Related on PULSE

• [How do you evaluate RAG retrieval quality?](/knowledge/ai0245)
• [How do you chunk documents for retrieval?](/knowledge/ai0248)
• [How do you choose an embedding model?](/knowledge/ai0251)
• [How do you cache LLM responses safely?](/knowledge/ai0253)
• [How do you monitor an AI pipeline in production?](/knowledge/ai0244)