What is the best way to cache embeddings at scale in 2027?

The outcome you should expect

When teams say "cache embeddings," they usually mean two different things that get tangled together, and separating them is the single largest determinant of whether the project succeeds. The first meaning is *avoid recomputing an embedding you already paid for*: you have a document, a query, or a support ticket, you sent it to an embedding model once, and you should never pay for that inference again. That is an exact-match cache, keyed on a deterministic hash of the input text plus the model identifier plus any normalization settings. The second meaning is *avoid rescanning your whole corpus to find similar vectors*: that is an approximate nearest-neighbor index, and it is a fundamentally different data structure with different cost, latency, and failure characteristics. A production system at scale almost always needs both, layered.

The outcome from getting the exact-match layer right is immediate and unambiguous. Embedding inference is the dominant recurring cost in most retrieval pipelines that re-index frequently, and any corpus with meaningful churn — a documentation site, a CRM note stream, a ticketing queue — regenerates the same text repeatedly. Content-addressed caching collapses that. If forty percent of your incoming documents are unchanged since the last crawl, forty percent of your embedding spend disappears the day you deploy a hash-keyed cache, and it disappears permanently rather than as a one-time optimization. The same applies on the query side: in a customer-facing search box, a small number of head queries account for a disproportionate share of traffic, and caching their vectors eliminates a network round trip to the embedding provider before the vector search even starts.

The outcome from the ANN layer is latency and unit economics rather than raw savings. A brute-force cosine scan over ten million 768-dimensional float32 vectors is roughly 30 GB of memory bandwidth per query — technically possible on a big machine, absurd as a per-request cost. An HNSW index over the same corpus answers in single-digit to low-double-digit milliseconds because it touches a tiny fraction of the graph. What you are buying is a bounded, predictable p99 that does not grow linearly with corpus size. What you are paying is memory, index build time, and a recall figure below 100 percent.

Realistically, a well-built two-tier cache on a mid-sized corpus (one to ten million vectors) should land you a p99 retrieval latency in the 5–20 ms range for the vector hop, a meaningful reduction in embedding API calls proportional to your content churn rate, and a memory footprint you can compute exactly in advance rather than discover in production. Those three numbers — latency, hit rate, footprint — are the ones to put on a dashboard before you write a line of integration code, because every architectural decision downstream trades one against the others.

One caveat worth stating up front: caching is not free correctness. An embedding cache is a correctness hazard the moment the model version changes, and the number of teams who have silently mixed vectors from two model versions in one index is large. The key design that prevents this is covered below, and it is the cheapest insurance in the whole system.

What drives that outcome

Four variables dominate: dimension, precision, index type, and churn rate. Everything else is second-order.

Dimension and precision set your floor. The memory arithmetic for raw vectors is exact and worth internalizing: vectors × dimensions × bytes-per-component. Ten million vectors at 768 dimensions in float32 is 10,000,000 × 768 × 4 ≈ 30 GB before any index overhead. The same corpus at 1,536 dimensions — the size several popular general-purpose embedding models emit — doubles to roughly 61 GB. Move to int8 scalar quantization and you divide by four, to about 7.5 GB at 768 dimensions. Product quantization compresses far more aggressively, at a larger recall cost. This is why the dimension choice made by a data scientist evaluating model quality quietly sets the infrastructure bill for the next two years, and why anyone shopping for the best cache should settle dimension first. Some newer embedding models support Matryoshka-style truncation, where you can keep a prefix of the vector and retain most of the retrieval quality — if your model supports that, truncating from 1,536 to 768 or 512 is the highest-leverage single change available.

Index type sets your latency-versus-memory curve. Flat (brute-force) indexes give exact results, need no build step, and scale linearly in query cost — perfectly fine below roughly one hundred thousand vectors, and genuinely the right answer for small tenant shards. HNSW builds a navigable small-world graph with two knobs that matter: M (connections per node, driving graph memory) and ef_construction (build-time search width, driving build cost and final quality). At query time ef_search trades latency for recall on a live, per-query basis, which is the knob you actually tune in production. IVF-family indexes partition the space into clusters and probe only the nearest few, which is cheaper to build than HNSW but usually needs more probes to reach comparable recall. Disk-resident indexes like DiskANN keep the bulk of the graph on SSD and hold only a compressed representation in memory, which is how you make hundreds of millions of vectors affordable.

Churn rate determines whether the exact-match tier even matters. If your corpus is static — a fixed reference library indexed once — the exact-match cache saves you almost nothing after the first build, and you should spend your effort on the index. If your corpus is a stream of near-duplicates, the exact-match tier is where nearly all the savings live.

The cache key deserves its own paragraph because it is where systems break. The key must include the model identifier and version, the output dimension, and the exact normalization applied to the input. Hash the concatenation with SHA-256 and use that as your key. The reason is a failure mode that is silent rather than loud: if you upgrade your embedding model and keep the old key scheme, you will serve a mix of old-model and new-model vectors from the same index. Cosine distances between vectors from different models are meaningless — not wrong in an obvious way, just noise — so retrieval quality degrades gradually while every health check stays green. Versioning the key makes an upgrade a cache miss instead of a corruption, which is exactly the behavior you want. It also gives you a clean rollback: the old vectors are still there under the old key prefix.

Benchmarks and realistic ranges

Published vendor benchmarks are run on tuned hardware with favorable parameters, so treat any single number as a ceiling rather than an expectation. The useful move is to know the shape of the ranges and then measure your own.

In-memory vector stores — Redis with the RediSearch/vector capability in Redis Stack, and Qdrant — sit at the fast end. For roughly a million 768-dimensional vectors with HNSW indexing on a well-provisioned node, single-digit-millisecond p99 for the search hop is a reasonable target, and both can be pushed to high query throughput on multi-core hardware. Redis's distinctive advantage is that it is probably already in your stack: you get vector search, the metadata, and your existing rate-limit and session keys in one system, plus hybrid queries that combine vector similarity with a metadata filter in a single FT.SEARCH call. Qdrant, written in Rust, offers payload indexing so metadata filters do not collapse ANN performance, and built-in quantization to cut memory. Both support real-time upserts, which matters enormously for churn-heavy corpora.

Managed services — Pinecone being the most widely deployed — trade a few milliseconds of latency for the elimination of sharding, replication, and index-tuning work. Expect low-double-digit p99 rather than single digits, largely because you are crossing a network boundary you do not control. Pinecone's serverless offering changed the cost model meaningfully by decoupling storage from query volume, which suits spiky or long-tail workloads far better than always-on provisioned capacity. The honest trade is that at very large scale, managed pricing is a multiple of self-hosted infrastructure cost, and you are paying that multiple for engineering time you would otherwise spend on operations.

Hybrid and general-purpose systems — pgvector on PostgreSQL, Elasticsearch's dense_vector with kNN search, Weaviate, Milvus, OpenSearch — occupy the middle. pgvector deserves special attention because it is the pragmatic default for a very large number of teams: if your documents already live in Postgres, adding a vector column and an HNSW index lets you join vector similarity against your relational data in one SQL statement, with your existing backups, replicas, permissions, and monitoring. Latency will be higher than a dedicated in-memory store — typically tens of milliseconds rather than single digits at a million vectors — but the operational savings are real and the ceiling is higher than skeptics assume. Elasticsearch is the analogous argument for teams already running it for logs or full-text search, with the bonus of genuinely good hybrid BM25-plus-vector ranking.

Libraries rather than services — FAISS from Meta is the reference implementation for large-scale similarity search and remains the fastest path to indexing enormous static corpora, particularly with GPU acceleration. It is a library, not a database: no server, no persistence layer, no upserts without an index rebuild. That makes it excellent for offline batch indexing and periodic full rebuilds, and poor for a live-updating corpus. A common and effective pattern is FAISS for the bulk cold index plus Redis for the hot, mutable layer. LanceDB fills a similar niche from the disk-first direction: an embedded, columnar, file-format-backed store that reads from local disk or object storage, trading latency for dramatically lower cost per vector.

For the exact-match tier itself, the numbers are simpler. A vector is a small blob — 3 KB at 768 float32 dimensions, 1.5 KB at float16, 768 bytes at int8 — and any key-value store handles it trivially. Redis, Memcached, or even a managed KV service will return one in sub-millisecond time on a warm connection. The interesting engineering question is not throughput but eviction and TTL policy, which is discussed below.

Set your own baseline before choosing. Take ten thousand real queries from your logs, embed them with your actual model, build the index with default parameters, and record p50/p95/p99 latency plus recall@10 against an exact brute-force ground truth on a sample. That measurement takes an afternoon and invalidates more vendor claims than any amount of reading.

Risks, edge cases, and failure modes

Model-version drift is the top risk and worth repeating because it is invisible. Every vector in a shared space must come from the same model, the same version, and the same pooling/normalization path. Version your keys and your index names. When you upgrade, build the new index alongside the old one, shadow-read against both, compare retrieval quality on a labeled set, then cut over and delete. Never mutate a live index in place across a model boundary.

Recall collapse under filters catches people mid-scale. HNSW is built to traverse a graph toward nearest neighbors; when you add a restrictive metadata filter — "only documents from this tenant, in this category, from the last thirty days" — a naive implementation either post-filters the top-k (returning far fewer results than requested, sometimes zero) or pre-filters and destroys the graph structure. Systems that handle this well implement filtered search natively at the index level. Test explicitly with your most selective realistic filter, not with an unfiltered query, because unfiltered benchmarks hide this entirely. This is also the strongest argument for tenant-sharded indexes in multi-tenant products: a per-tenant index makes the most common filter free.

Unbounded cache growth is mundane and expensive. An exact-match embedding cache with no eviction grows monotonically with every unique string your system has ever seen, including one-off typo'd queries that will never recur. Use an LRU or LFU policy on the query-side cache with a TTL measured in days, and keep the document-side cache long-lived because documents genuinely recur. Distinguishing those two populations — high-cardinality, low-recurrence queries versus low-cardinality, high-recurrence documents — and giving them different policies is the difference between a cache that costs a few gigabytes and one that costs a few hundred.

Cache stampede on cold start hits when a deploy flushes memory and every request simultaneously misses and calls the embedding API. The mitigations are standard distributed-systems hygiene: request coalescing (single-flight per key, so N concurrent misses on the same key trigger one inference), a small jitter on TTLs so entries do not expire in lockstep, and a warming pass that preloads head queries after a restart. Rate limits on hosted embedding APIs make this worse than it sounds, because the stampede does not just cost money, it produces a wave of 429s and a latency cliff.

Silent staleness is the failure mode CLAUDE.md's own operational law exists to prevent, and it applies squarely here. A re-embedding job that dies quietly leaves an index that is technically healthy — it responds, it returns results — while drifting further from the source corpus every day. Monitor freshness explicitly: track the maximum age of the newest indexed document, alert if the newest document is older than your expected ingest interval, and record the count of vectors in the index against the count of rows in the source of truth. A cache that returns fast wrong answers is worse than one that returns slow right ones, because nothing pages you.

Quantization recall loss is real but usually acceptable, and the mistake is measuring the wrong thing. Scalar int8 quantization typically costs a small amount of recall for a 4× memory reduction; product quantization costs more for far greater compression. The correct evaluation is not "recall@10 against exact search" but "does the end-to-end answer quality change" — because most retrieval feeds a reranker or an LLM that is tolerant of a slightly reordered candidate set. Measure downstream. If a cross-encoder reranker sits between retrieval and output, you can often quantize aggressively and let the reranker repair the ordering, buying most of the memory savings for almost no user-visible cost.

Normalization mismatch is the quiet one. If your ingest path lowercases and strips punctuation but your query path does not, your cache hit rate craters and your embeddings sit in slightly different regions of the space. Put normalization in one shared function, call it from both paths, and include a hash of its version in the cache key. The same applies to distance metric: cosine is standard for sentence embeddings, dot product is equivalent on unit-normalized vectors and faster, and Euclidean behaves differently on unnormalized ones. Fix the metric at index creation and record it in your config, because a mismatch between how you normalized and what metric you chose is a subtle quality bug that never throws an error.

Multi-tenant isolation deserves a line. Sharing one index across tenants with a tenant-id filter is efficient until it is a security incident. If tenant data is sensitive, use physically separate indexes or collections; most of the mature vector stores support multi-tenancy as a first-class concept precisely because filter-based isolation is a thin guarantee.

A practical rollout plan

Sequence matters more than tool choice. Teams that pick the store first and design the keys later spend the following quarter migrating.

Week one — measure and decide the numbers. Pull real traffic: how many documents, what churn rate per day, how many queries per second at peak, what fraction of queries are repeats. Compute the memory arithmetic for your dimension at float32, float16, and int8. Decide your latency budget for the retrieval hop specifically, separate from the LLM generation time that will dwarf it. If your end-to-end budget is two seconds and generation takes 1.4 of it, an extra 15 ms of vector latency is irrelevant and you should optimize for cost instead — a conclusion that changes the recommended architecture entirely.

Week two — build the exact-match tier first. It is simpler, it delivers savings immediately, and it is independent of which vector store you eventually choose. Implement the versioned key, the shared normalization function, single-flight coalescing, and separate TTL policies for query and document caches. Instrument hit rate from day one, split by population. This tier alone often justifies the whole project.

Week three — stand up the ANN index with defaults and benchmark honestly. Start with the store your team already operates: Postgres shops start with pgvector, Redis shops with Redis Stack, Elastic shops with kNN search. Only move to a dedicated store if the benchmark says you must. Build with default HNSW parameters, run your ten-thousand-query harness, record p50/p95/p99 and recall@10 against brute-force ground truth on a sample. Then tune one knob at a time.

Week four — add the operational skin. Freshness monitoring, index-size-versus-source-count reconciliation, a documented rebuild procedure, and a shadow-index path for model upgrades. Then test the failure modes deliberately: flush the cache under load and watch for a stampede, run your most selective filter and check recall, kill the ingest worker and confirm the freshness alert fires within one interval.

Two adjacent practices are worth folding in while you are here, because they share infrastructure. The first is semantic caching of LLM responses: once you have an embedding cache and a vector index, caching whole answers keyed by query similarity is a small increment of work and a large cost reduction on repetitive question streams. Set the similarity threshold conservatively — a threshold too loose returns a confidently wrong cached answer to a subtly different question, which is a far worse failure than a cache miss. The second is batching on the ingest side: embedding APIs are dramatically more efficient per token when you send batches rather than single strings, and a queue that accumulates misses for fifty milliseconds before flushing a batch often cuts ingest cost and wall-clock time substantially with no user-visible latency change.

Finally, keep a rebuild runbook and rehearse it. Every embedding system eventually needs a full re-index — a model upgrade, a dimension change, a normalization fix, a corrupted shard. The teams that handle this calmly are the ones who have already scripted it, know how long it takes, know what it costs, and can run it against a shadow index without touching production traffic. The ones who have not scripted it discover during an incident that a full rebuild takes eleven hours and there is no way to serve traffic in the meantime.

Related questions

Should I cache embeddings in the same store I use for vector search?

Usually yes for simplicity — Redis, Qdrant, and pgvector all store the vector and metadata together. Split them only when the exact-match cache has a very different access pattern or eviction policy than the searchable index, which happens most often with high-cardinality query caches.

How do I invalidate an embedding cache when the source document changes?

Key on a hash of the content itself rather than the document ID. Changed content produces a different hash, which is automatically a miss, and the old entry ages out via TTL. This is content-addressed invalidation and it removes an entire class of stale-cache bugs.

Is it worth caching query embeddings, not just document embeddings?

Yes when query traffic is head-heavy. A search box where a small set of queries dominates gets a high hit rate on a modest cache, saving an API round trip before search begins. For long-tail, mostly-unique queries the hit rate is low and the memory is better spent elsewhere.

Does quantization break my existing similarity thresholds?

Often, subtly. Quantized distances shift slightly, so a hard-coded threshold tuned on float32 may admit or reject differently. Re-tune thresholds against the quantized index on a labeled set rather than assuming they transfer.

Can one index hold embeddings from two different models?

No. Distances between vectors from different models are meaningless. Use separate indexes, or a model-version prefix in the key and a filter, and cut over via a shadow index after comparing quality.

FAQ

How much memory do 10 million embeddings actually need?

Compute it directly: vectors × dimensions × bytes-per-component. Ten million 768-dimensional float32 vectors is about 30 GB of raw vector data, plus index overhead — HNSW adds meaningfully on top depending on your M setting. At 1,536 dimensions it roughly doubles. Int8 scalar quantization cuts the raw portion by about 4×, to roughly 7.5 GB at 768 dimensions. Always budget the index overhead separately; a common mistake is provisioning for the raw vectors only and running out of headroom during the graph build, which is the most memory-intensive moment in the system's life.

What is a good cache hit rate to aim for?

There is no universal target, because it depends entirely on your content churn and query distribution. The useful framing is comparative: measure hit rate separately for the document population and the query population, then ask whether each matches the recurrence you expect from that population. A document cache with a low hit rate on a corpus you believe is mostly static means your normalization or keying is broken, not that caching does not help. A query cache with a low hit rate on genuinely unique long-tail queries is working correctly and simply is not worth much memory.

Redis or a dedicated vector database?

If you already run Redis and your corpus fits comfortably in memory, Redis Stack is an excellent answer — you get vector search, metadata, and hybrid filtering in a system your team already knows how to operate, back up, and monitor. Move to a dedicated store like Qdrant, Weaviate, or Milvus when you need capabilities Redis does not prioritize: aggressive built-in quantization, disk-resident indexes for cold data, first-class multi-tenancy, or clustering topologies designed specifically around vector workloads.

When does pgvector stop being enough?

Later than most people expect. pgvector with an HNSW index handles millions of vectors respectably and gives you transactional consistency between your vectors and your relational data, which is a genuine advantage nothing else offers. The pressure points are memory contention with your OLTP workload on the same instance, index build time on very large tables, and the operational awkwardness of vacuuming a table with a large vector index. Watch your p99 and your build times; when either becomes the constraint, a read replica dedicated to vector queries buys you significant runway before a migration.

Do I need a GPU for this?

For serving, almost never — ANN search is memory-bandwidth-bound, not compute-bound, and CPU HNSW is fast. For building indexes over very large static corpora, GPU acceleration via FAISS makes a large difference in wall-clock rebuild time. And for generating the embeddings themselves, if you self-host the model rather than calling an API, a GPU is what makes batch throughput practical. Separate the three questions; they have different answers.

How should I handle a model upgrade without downtime?

Build a second index under a new versioned name while the old one serves traffic. Re-embed your corpus into it, using the exact-match cache keyed on the new model version so restarts do not repeat work. Shadow-read: send a sample of live queries to both indexes and compare results against a labeled relevance set. When the new index wins, flip the read path via config, keep the old index for a rollback window, then delete it. The cost is temporarily double storage; the benefit is that a bad upgrade is a config flip away from being undone.

Sources

• Redis vector search documentation
• pgvector extension for PostgreSQL
• FAISS repository and wiki
• Qdrant documentation
• Weaviate developer documentation
• Milvus architecture overview
• Elasticsearch kNN search reference
• Pinecone documentation
• HNSW paper — Malkov & Yashunin
• DiskANN research page (Microsoft Research)

Related on PULSE

• What is a semantic cache and how much can it cut inference costs?
• How do you evaluate LLM output quality at scale?
• How do you scale LLM inference to handle thousands of concurrent users?
• How do you choose an embedding model for retrieval?