What is a semantic cache and how much can it cut inference costs?

What it is and why it matters

Traditional caching is a string comparison. You hash the prompt, look it up, and either hit or miss. That works fine for a REST endpoint where /users/42 is always literally /users/42, and it fails almost completely for natural language, because humans never ask the same question the same way twice. "How do I reset my password?" and "I forgot my password, what now?" and "password reset help" are three distinct strings that hash to three distinct keys, so an exact-match cache returns three misses and you pay for three full inference calls that produce three near-identical answers.

A semantic cache replaces the hash with a distance measurement. Every incoming prompt is converted into a dense vector by an embedding model — a list of numbers, commonly 384, 768, or 1,536 dimensions depending on the model you choose. Those numbers encode meaning rather than characters, so the three password questions above land close together in vector space even though they share few words. The cache runs an approximate nearest-neighbor search against previously stored vectors, and if the closest stored prompt is within a configured similarity threshold, it returns that stored response instead of calling the model.

The economics are what make this worth building. An embedding call is roughly three orders of magnitude cheaper than a chat-completion call on the same text, because embedding models are far smaller and they emit a fixed-size vector rather than generating tokens one at a time. Generation is the expensive half of inference: every output token requires a full forward pass through the model. A cached hit skips all of it. You pay for one small embedding, one vector search, and a database read.

The latency story matters as much as the cost story, and in user-facing products it often matters more. A large model producing a few hundred tokens takes somewhere between one and ten seconds depending on model size, prompt length, and how loaded the provider is. A cache hit returns in single-digit to low-double-digit milliseconds. That is not a marginal improvement, it is a categorical one — the difference between a spinner and an instant answer. Teams that adopt semantic caching for cost reasons frequently keep it for the responsiveness.

Where this pays off is any workload with a fat head of repeated intent. Customer support bots are the canonical case, because a support corpus is overwhelmingly the same forty questions asked ten thousand different ways. Internal documentation assistants behave the same way — new engineers all ask how to get database access in their first week. Product FAQ widgets, onboarding flows, compliance Q&A, and pre-sales chat all share the shape: high volume, narrow intent space, tolerant of a canonical answer. The inverse is also worth naming plainly. Creative writing tools, personalized recommendation engines, and anything conditioned heavily on per-user state have almost no semantic repetition, and a cache in front of them will burn embedding calls and return nothing. The technique is not universally applicable and pretending otherwise is how teams end up disappointed.

There is also a quality argument that gets underrated. When the cache serves a stored answer, every user asking that question gets the identical response. For a support bot, consistency is a feature — you have effectively promoted one vetted answer to canonical status. Some teams exploit this deliberately by pre-seeding the cache with human-reviewed answers to their top questions, which converts the cache from a pure cost optimization into a lightweight content-governance layer. That hybrid is often the strongest version of the pattern: the cache guarantees your best answers get served, and the model handles the long tail.

The step-by-step process

The runtime flow is short enough to hold in your head, but every step has a decision attached to it.

Normalize the incoming prompt. Strip whitespace, lowercase if your embedding model is case-sensitive in ways you do not want, and — critically — separate the variable part of the prompt from the fixed scaffolding. If your application wraps every user question in a 600-token system prompt, embedding the whole assembled string is a mistake: the shared scaffolding dominates the vector and makes everything look similar to everything else. Embed the user's actual question only.

Embed it. One call to a small embedding model. This can be a hosted API or a local sentence-transformer running on CPU. Local models in the MiniLM family produce 384-dimension vectors in a few milliseconds on commodity hardware and cost nothing per call, which matters because you pay this cost on every request including misses. Hosted embedding APIs are more accurate on nuanced text but add a network round trip and a per-token charge.

Search. Query your vector store for the nearest neighbor, usually by cosine similarity. Almost all production stores use an approximate index — HNSW is the common choice — because exact search over millions of vectors is too slow. Approximate search trades a small recall loss for orders-of-magnitude speedup, and at cache scale that trade is nearly free.

Threshold. Compare the top result's similarity score against your cutoff. Above it, you have a hit. Below it, a miss. This single number is the most consequential configuration in the entire system and the section below on failure modes is largely about getting it right.

On a hit, return and log. Serve the stored response, record the hit, and — this is the step people skip — record which stored entry was served and what the incoming query was. Without that log you cannot audit false positives later.

On a miss, call the model, then write back. Run normal inference, return the response to the user, and asynchronously store the new prompt vector plus the response in the cache. Do the write after the response has been streamed to the user so caching never adds latency to the miss path.

A few implementation details separate a demo from something you can run in production. Scope your cache keys — a single global cache is almost always wrong. Partition by tenant, by language, by model version, and by any system-prompt variant that materially changes the answer. Two users at different companies asking "what's our refund policy" deserve different answers, and a naive shared cache will happily serve one company's policy to the other. That is not a performance bug, it is a data-leak incident.

Version your entries. Store the model name, the embedding model name, and the prompt-template version alongside every cached response. When you upgrade the underlying model, you need the ability to invalidate everything generated by the old one, and if you swap embedding models you must rebuild the index from scratch — vectors from different embedding models are not comparable, and mixing them silently produces garbage similarity scores rather than an error.

Make writes asynchronous and make failures non-fatal. If the vector store is down, the correct behavior is to treat every request as a miss and keep serving users from the model. A cache that takes down your product when it fails has negative expected value regardless of how much it saves.

Costs, timelines, and typical ranges

The savings math is simple enough to do on a napkin, and doing it before you build is the single highest-leverage hour in this project.

Your cost reduction is approximately your cache hit rate, minus the embedding overhead you now pay on every request. Because embedding is so much cheaper than generation, that overhead is usually a rounding error — on a short query, an embedding call costs a tiny fraction of a cent while the generation it replaces costs meaningfully more. So a 50% hit rate is roughly a 50% reduction in inference spend, and a 70% hit rate is roughly a 70% reduction. The published ranges you see quoted — 30% to 70% — are really just the observed range of hit rates across different workloads, not a property of any particular caching tool.

Hit rate is determined by your traffic, not by your vendor. Realistic bands look roughly like this. A mature customer-support bot with a narrow product surface lands high, often 50–70%, because the question distribution is brutally head-heavy. An internal knowledge assistant sits in the middle, maybe 30–50%, with a strong weekly rhythm — Monday morning is repetitive, Thursday afternoon is long-tail. A general-purpose assistant with an open-ended prompt space lands low, often under 20%, and may not clear the operational overhead. A creative or personalization workload lands near zero. Before writing code, sample a few thousand real production prompts, embed them offline, and cluster them. If fewer than a quarter fall into tight clusters, a semantic cache is not your cost lever and you should look at model routing or prompt compression instead.

On infrastructure cost: the cache itself is cheap relative to what it saves. A few million cached entries at 768 dimensions is a few gigabytes of vectors plus the response text, which is a modest instance on any managed vector store or a self-hosted Redis, Qdrant, Milvus, or pgvector deployment. The recurring cost is typically one to two orders of magnitude below the inference bill it offsets. That ratio holds across essentially every option, which is why tool choice should be driven by latency requirements and operational fit rather than by price comparison.

On latency budgets: cache hits typically land in the 1–50ms range end to end, with the spread driven mostly by where the embedding happens. A local embedding model plus an in-process index is at the fast end. A hosted embedding API plus a network call to a managed vector database is at the slow end — still fifty to a hundred times faster than generation. On misses you pay that same overhead on top of normal inference, so a poorly-implemented cache with a slow hosted embedding call can add real latency to every miss. If your hit rate is low, that overhead is pure loss, which is another argument for measuring hit rate before committing.

On timelines: a working prototype using an existing framework's cache integration is an afternoon. A production deployment — with tenant scoping, TTLs, model versioning, hit-rate telemetry, false-positive sampling, and a kill switch — is realistically two to four weeks of one engineer's time, and most of that is not the cache itself but the observability and the threshold tuning. Budget a further two to four weeks of watching real traffic before you trust the numbers, because early hit rates are misleading. A cold cache starts at 0% and climbs; the curve typically flattens after a few thousand distinct queries, and whatever it flattens at is your real number.

There is a related lever worth naming, because teams often conflate the two. Provider-side prompt caching — where the model host caches the KV state of a long, repeated prefix — is a different mechanism with a different payoff. It reduces the cost of re-processing a large shared system prompt or document, not the cost of generation, and it applies on exact prefix matches rather than semantic ones. The two compose well: prompt caching cuts the input cost of your misses, semantic caching eliminates a chunk of the calls entirely. If you have both a long system prompt and repetitive user intent, run both. They stack.

Where teams get it wrong

Threshold tuning by vibes. The similarity cutoff is the whole system and most teams set it to a round number from a tutorial and never revisit it. Set it too low and you serve confidently wrong answers — "how do I cancel my subscription" matches "how do I change my subscription," and a user who wanted to leave gets told how to upgrade. Set it too high and you get a cache that only catches near-identical strings, which is an exact-match cache with extra latency. The correct process is empirical: pull a few hundred real query pairs, have a human label whether each pair should share an answer, then plot precision and recall across candidate thresholds and pick from the curve. This takes a day and it is the difference between a cache that works and one that quietly degrades your product.

Treating false positives as a rounding error. A cache miss costs money. A false positive costs trust, and it is invisible in your metrics unless you deliberately look for it. Hit rate goes up and cost goes down, so the dashboard says success while users get subtly wrong answers. Build a sampling audit from day one: log a random slice of hits with the incoming query, the matched query, the score, and the served response, and have a human review a batch weekly. Cheaper alternative: for scores in an uncertain band just above your threshold, run the miss path anyway on a small percentage and compare the fresh answer to the cached one.

Negation and small-word blindness. Embedding models compress meaning, and short function words carry disproportionate semantic weight relative to their embedding contribution. "Can I deduct this expense" and "Can I not deduct this expense" can score alarmingly close. So can pairs differing only in a number, a date, a currency, or a proper noun — "what's the limit for a Pro plan" versus "what's the limit for an Enterprise plan." This is the most dangerous failure class because the queries genuinely are semantically similar; they are just not answer-equivalent. Mitigations: extract entities and numbers from the query and require exact match on them as a filter alongside the vector search, or run a cheap cross-encoder re-rank on the top candidate before accepting the hit.

No expiry strategy. Cached answers age. A pricing question cached in January is wrong in March. A policy answer cached before a rewrite is now actively misleading. Every entry needs a TTL calibrated to how fast its subject changes, and you need event-driven invalidation for the cases where content changes on a known trigger — when the docs page updates, purge everything derived from it. Tag entries with their source documents at write time so you can do that purge surgically instead of nuking the whole cache.

Ignoring conversational context. In a multi-turn chat, "what about the second one?" is meaningless in isolation and dangerously ambiguous when embedded alone. Either exclude follow-up turns from caching entirely, or embed a context-resolved rewrite of the query rather than the raw text. Many teams already run a query-rewriting step for retrieval; reuse its output as the cache key and both problems collapse into one.

Cross-tenant and cross-user leakage. Covered above but it bears repeating as a failure mode because it is the one that ends careers. Any cache that can serve one customer's data to another is a security incident, not a bug. Namespace aggressively, test the isolation explicitly, and never cache anything derived from user-private context in a shared namespace.

Skipping the cold-start problem. A fresh cache saves nothing, which makes the first week look like a failure and gets projects killed prematurely. Pre-warm it. Take your top few hundred historical questions, generate answers offline, review them, and seed the cache before launch. You start at a useful hit rate on day one and you get the content-governance benefit for free.

Caching the wrong layer. In a retrieval-augmented pipeline you have several cacheable stages: the retrieval results, the reranked context, and the final generated answer. Caching only the final answer is coarse — a small change in the question misses entirely and you re-run the whole pipeline including the expensive retrieval. Caching retrieval results separately gives you partial credit on near-misses, and in RAG systems that intermediate cache is often the better cost lever because retrieval and reranking are themselves not free.

Decision framework: when to choose what

Start with the question of whether to cache at all, then move to how.

If your sampled traffic shows under roughly 20% clusterable repetition, stop. Your cost lever is elsewhere — route easy queries to a smaller model, compress your system prompt, or batch offline work. A cache on non-repetitive traffic is negative value.

If repetition is high and your answers are stable, cache the final response and consider pre-seeding it with reviewed content. If repetition is high but answers depend on freshly-retrieved data, cache the retrieval layer rather than the generation, and let the model synthesize fresh each time. If repetition is high but answers are user-specific, cache per-user namespace only, and accept a lower hit rate as the price of correctness.

On storage: if you already run Redis, use it — RedisVL adds vector indexing to infrastructure your team already operates, and operational familiarity beats a marginally better benchmark every time. If you already run Postgres and your volume is modest, pgvector keeps the cache in a database you already back up and monitor. If you need a purpose-built store at scale, Qdrant, Milvus, Weaviate, and Chroma all serve this well and differ more in operational model than in capability. If you want zero infrastructure, a managed vector service like Pinecone removes the ops burden entirely at a higher per-query cost. GPTCache remains the reference open-source implementation of the whole pattern and is worth reading even if you build your own, because its modular split of embedder, storage, and evaluator is the right decomposition.

On the similarity evaluator: a bare cosine threshold is the fast default. Add a cross-encoder re-rank on borderline scores when false positives are expensive — legal, medical, financial, or anything where a wrong answer has consequences beyond mild annoyance. The re-rank costs more per hit but it is still vastly cheaper than generation, and it converts your riskiest band of matches into confident ones.

Whatever you choose, ship the telemetry with version one, not version two. You need hit rate, score distribution, cost saved, latency on both paths, and a sampled false-positive review. Without those five numbers you cannot tune the threshold, and an untuned threshold is the failure mode that makes semantic caching look like a bad idea when it is actually a very good one that was configured carelessly.

Related questions

Does a semantic cache work with streaming responses?

Yes, with a caveat. Store the complete response text on a miss, then replay it token-by-token on a hit to preserve the streaming UX. The replay is synthetic — you already have the full text — so it can be paced arbitrarily or delivered instantly.

How does semantic caching differ from provider prompt caching?

Provider prompt caching reuses the model's internal state for a repeated exact prefix, cutting input-processing cost on calls you still make. Semantic caching eliminates the call entirely on meaning-equivalent queries. Different mechanisms, different savings, and they stack cleanly.

Can I use a semantic cache with a self-hosted model?

Yes. The cache sits in front of any inference endpoint and never inspects the model. With self-hosted models the savings show up as reclaimed GPU capacity and lower queue depth rather than a smaller API bill, which is often the more valuable currency.

What similarity threshold should I start with?

Cosine similarity in the 0.85–0.95 band is a common starting point, but treat it as a placeholder. Label a few hundred real query pairs, plot precision and recall across candidate values, and pick from your own curve rather than a default.

Should the cache be shared across tenants?

Almost never for anything derived from tenant data. Namespace per tenant by default. Only share a namespace for genuinely public, identical-for-everyone content, and test the isolation boundary explicitly before launch.

FAQ

How much can a semantic cache actually cut inference costs?

Roughly as much as your cache hit rate, since embedding overhead is negligible next to generation. Commonly cited ranges run 30–70%, but that band reflects observed hit rates across different workloads rather than any tool's capability. Support bots land high, open-ended assistants land low, creative tools land near zero. Measure your own traffic's repetition before assuming a number.

Does the cache add latency to queries that miss?

Yes — one embedding call plus one vector search, typically a few milliseconds with a local embedding model and a few tens of milliseconds with a hosted one. Against multi-second generation that is negligible on hits and a small tax on misses. Keep writes asynchronous so storing the new entry never blocks the response.

What happens when I upgrade the underlying model?

Cached responses were generated by the old model, so decide deliberately whether they remain acceptable. Store the model version on every entry so you can invalidate selectively. Separately, if you change the *embedding* model you must rebuild the index entirely — vectors from different embedding models are not comparable and mixing them produces meaningless scores rather than errors.

How do I stop stale answers from being served?

Give every entry a TTL matched to how fast its subject changes — hours for anything pricing- or availability-related, days to weeks for stable policy and how-to content. Tag entries with the source documents they derive from so a content update can purge exactly the affected entries instead of clearing the whole cache.

Is this the same as caching RAG retrieval results?

No, though both are worth doing. Retrieval caching stores the documents fetched for a query; semantic caching stores the final generated answer. In RAG pipelines the retrieval cache often delivers more value because it still earns partial credit on near-miss queries, where an answer-level cache misses completely and re-runs the whole pipeline.

How do I detect false positives before users complain?

Sample. Log a random slice of hits with the incoming query, the matched entry, and the similarity score, and have a human review a batch on a schedule. For scores just above your threshold, occasionally run the model anyway and compare its answer to what the cache served. Divergence there tells you your threshold is too loose.

Sources

• GPTCache — open-source semantic cache for LLMs
• Redis vector search documentation
• LangChain LLM caching integrations
• Qdrant vector database documentation
• Milvus documentation
• Weaviate developer documentation
• pgvector — vector similarity search for Postgres
• FAISS wiki — index types and guidelines
• Sentence-Transformers documentation
• Anthropic prompt caching documentation

Related on PULSE

• [What causes high latency in LLM inference and how do you fix it?](/knowledge/ai389)
• [What is the difference between batch and real-time inference infrastructure?](/knowledge/ai409)
• [What is the best way to cache embeddings at scale?](/knowledge/ai419)
• [How do you choose an inference accelerator: GPU, TPU, or custom silicon?](/knowledge/ai415)
• [The 10 Best LLM Inference Servers in 2027](/knowledge/ai342)