What is the best way to store embeddings for production RAG in 2027?

What it is and why it matters

An embedding is a fixed-length array of floating-point numbers — commonly 384, 768, 1024, 1536, or 3072 dimensions depending on the model — that represents a chunk of text in a geometric space where semantically similar chunks land near each other. Storing embeddings for production RAG means solving four problems at once: persisting the vectors durably, indexing them so nearest-neighbor search returns in tens of milliseconds instead of seconds, keeping the metadata that lets you filter and cite results, and keeping all three in sync as documents change.

The naive version — load every vector into a NumPy array in process memory and brute-force cosine similarity on each query — works fine up to roughly 50,000 to 100,000 vectors and is genuinely the correct choice at that scale. A brute-force scan over 100,000 vectors at 768 dimensions is about 300 MB of float32 math per query, which a modern CPU handles in double-digit milliseconds. Teams reach for infrastructure long before they need it, and then spend the next quarter operating it.

The reason storage choice matters more than it looks is that the vector is never the whole record. A production RAG retrieval almost always needs to answer "find the ten most similar chunks, but only from documents this user is permitted to see, only from the current version, and only from the last eighteen months." That is a filtered nearest-neighbor query, and how a given store implements filtering is the single largest source of surprise in production. Pre-filtering narrows the candidate set before the ANN search and gives exact results but can degrade to a scan; post-filtering searches first and drops non-matching results afterward, which is fast but can return four results when you asked for ten. Some engines do a hybrid where the filter is evaluated during graph traversal. You need to know which one you have.

The second reason it matters is that embeddings are derived data with a shelf life. When you change embedding models — and you will, because models improve — every vector you have stored becomes incompatible with every new query vector. There is no partial migration: a 1536-dimension OpenAI vector and a 1024-dimension Cohere vector do not live in the same index, and even two versions of the same model produce coordinates that are not comparable. The storage layer has to make full re-embedding a routine operation rather than an outage, which in practice means you keep the raw chunk text somewhere you can re-read cheaply, and you design for building a second index alongside the live one.

Anchoring on the word best is misleading here. There is no store that wins on every axis. What exists is a small set of well-understood architectures, each of which is the best choice for a recognizable situation, and a set of operational practices that matter more than which logo is on the database.

The step-by-step process

Building the storage layer is a sequence, and most teams that struggle skipped step one or step six.

Step 1 — Fix your chunking and metadata schema before you index anything. Chunk size drives everything downstream. Common production settings land between 256 and 1024 tokens per chunk with 10–20% overlap; smaller chunks retrieve more precisely but lose context, larger chunks retrieve fuzzier but give the generator more to work with. Decide the metadata fields now: document ID, chunk ordinal, source URI, tenant or ACL identifier, content hash, embedding model name and version, and an ingestion timestamp. The content hash is what lets you skip re-embedding unchanged chunks later, and the model version field is what makes a migration tractable. Retrofitting these onto a live index means a full rebuild.

Step 2 — Pick the embedding model and record its dimensionality as a schema constraint. Most vector stores require you to declare dimensionality at collection creation. Write the model identifier into the collection name or a required metadata field so a mismatched vector fails loudly at write time rather than silently producing garbage similarity scores.

Step 3 — Batch the embedding calls. Embedding APIs accept arrays; sending one chunk per request is the difference between an hour and a day for a large corpus. Batch sizes of 64–256 chunks per call are typical. Build in retry with exponential backoff and persist the resulting vectors to durable storage — object storage as Parquet or a plain table — before you index them. This is the single most valuable habit in the whole pipeline: if your vectors exist as files, rebuilding an index is a bulk load rather than a re-embedding bill.

Step 4 — Load into the index and choose the index type deliberately. HNSW is the default for good reason: it gives high recall at low latency and supports incremental inserts. Its knobs are m (graph connectivity, typically 16–64) and ef_construction (build-time search width, typically 100–400); higher values mean better recall and a slower, larger build. At query time, ef_search trades latency for recall directly and is the knob you tune in production. IVF-based indexes partition the space into lists and probe a subset, which uses far less memory but requires a training step and degrades when the data distribution shifts. Flat indexes do exact search and are correct at small scale.

Step 5 — Measure recall against ground truth before you trust it. Take a sample of a few thousand queries, compute exact nearest neighbors by brute force, and compare against what your index returns. Recall@10 below roughly 0.95 usually means your ef_search or probe count is too low. Teams routinely deploy an ANN index without ever checking this and then blame the language model for bad answers that were actually bad retrieval.

Step 6 — Wire the update path. Decide how a changed document propagates: delete-then-insert by document ID is the simplest correct pattern. Note that HNSW deletes are usually soft — the node is marked and skipped, and space is reclaimed only on compaction — so a high-churn corpus needs periodic rebuilds or your index grows without bound.

Costs, timelines, and typical ranges

The cost of storing embeddings has three components, and teams almost always underestimate the second and third.

Raw vector footprint is arithmetic you can do on a napkin. A float32 vector at 1536 dimensions is 6,144 bytes, roughly 6 KB. One million such vectors is about 6 GB of raw data. At 768 dimensions it is about 3 KB per vector and 3 GB per million. Add the HNSW graph on top: the graph edges typically add somewhere in the range of 25–60% overhead depending on your m setting, so budget closer to 8–10 GB per million 1536-dimension vectors in a memory-resident index. Metadata and the original chunk text often exceed the vectors themselves — a 500-token chunk is roughly 2 KB of UTF-8, so a million chunks is another 2 GB before indexes on the metadata columns.

Quantization changes this dramatically and is the most underused lever available. Scalar quantization to int8 cuts memory roughly 4× with a typically small recall cost that you can measure and often recover by re-ranking the top candidates against full-precision vectors. Binary quantization cuts it about 32× and is aggressive enough that re-ranking becomes mandatory, but for large corpora it converts an infeasible memory bill into a routine one. Several embedding models now support Matryoshka-style truncation, where you can simply cut a 3072-dimension vector down to 512 or 768 dimensions and retain most of the retrieval quality — this is a one-line change with a 4–6× storage reduction and should be the first thing you test.

Embedding compute is a one-time cost per chunk plus a recurring cost on updates and re-embeddings. Hosted embedding APIs are priced per million tokens and are among the cheapest calls in the LLM ecosystem — orders of magnitude below generation. Self-hosting a small open-weight embedding model on a single GPU is entirely viable and removes per-token billing, at the cost of running the GPU. The decision usually turns on volume: intermittent ingestion favors the API, continuous high-volume ingestion favors self-hosting.

The recurring infrastructure cost is where the surprises live. A memory-resident index has to fit in RAM, which means your bill is set by peak corpus size, not by query volume. Managed vector services typically bill on a combination of storage, provisioned compute, and read/write operations, and the pricing models differ enough between vendors that a spreadsheet comparison at your actual corpus size and query rate is worth an afternoon. Disk-backed indexes trade latency for cost and are the right answer for large, infrequently queried corpora.

Timelines. A prototype RAG index over a few thousand documents is an afternoon. A production storage layer with filtering, access control, incremental updates, and a tested re-embedding path is typically two to six weeks of engineering, with most of the time going into the update path and the ACL model rather than the search itself. Budget a full day to re-embed and rebuild for every few million chunks when you change models, and run that drill once before you need it.

Where teams get it wrong

Treating the vector store as the source of truth. The index is derived data. If you cannot rebuild it from your documents and your stored chunk text, you have a single point of failure that no backup strategy fully covers, because a corrupted or subtly-wrong index does not announce itself. Keep the chunks and the raw vectors in durable storage and treat the serving index as disposable.

Ignoring the filter path until it breaks. The most common production failure is a query that filters to a narrow tenant or date range and returns far fewer results than requested, or takes 40× longer than the unfiltered version. Test filtered queries at the selectivity you actually see in production — including the worst case where the filter matches 0.1% of the corpus — before launch.

Skipping hybrid search. Dense vectors are bad at exact matches: product SKUs, error codes, person names, acronyms. A pure semantic index will confidently return something adjacent to the error code you searched for instead of the error code itself. Combining BM25 or another lexical signal with vector similarity — typically fused with reciprocal rank fusion — reliably beats either alone, and most mature stores support it natively now. If your store does not, running a lexical index alongside and fusing at the application layer is straightforward.

Never measuring recall. Approximate search is approximate. Deploying an ANN index without a ground-truth comparison means you have no idea whether your retrieval quality problem is the embedding model, the chunking, or an ef_search value that is silently dropping the right answer.

Over-engineering at small scale. Standing up a distributed vector cluster for 20,000 chunks is a common and expensive mistake. Below roughly 100,000 vectors, brute force in a single process or a pgvector table with a modest index is faster to build, easier to debug, and often faster to query than the alternative.

Under-engineering the multi-tenant boundary. If one tenant's chunks can surface in another tenant's results, that is a data breach, not a relevance bug. Enforce tenancy structurally — separate collections or namespaces per tenant when tenant count is manageable, or a mandatory filter enforced server-side that the application layer cannot accidentally omit. Never rely on the caller to remember to pass the filter.

Forgetting soft deletes accumulate. In HNSW-based stores, deleted vectors are typically tombstoned rather than removed. A corpus with heavy churn will see memory grow and recall drift until someone runs a compaction or rebuild. Schedule it.

Decision framework: when to choose what

The honest answer is that four architectures cover nearly every production case, and the choice falls out of three questions: how many vectors, how complex the filters, and whether you already run a database you trust.

Under ~100,000 vectors, with an existing relational database: use a vector extension on that database. pgvector on PostgreSQL is the reference example. You get transactional consistency between your documents and their embeddings, real SQL filtering with real query planning, one backup story, and one on-call rotation. The joint filter-and-search problem largely disappears because the planner handles it. This covers a large majority of internal-tool and mid-market RAG applications, and teams that start here rarely need to leave.

100,000 to roughly 10 million vectors, with complex filtering or high update rates: a dedicated vector database earns its keep. This is the range where HNSW tuning, native hybrid search, quantization support, and purpose-built filtered-search implementations start to matter materially. Self-hosted open-source options and managed services both work; the deciding factor is usually whether you want to operate a stateful memory-heavy service.

Above ~10 million vectors, or with a hard cost ceiling: quantization and disk-backed indexing stop being optional. Look at binary or scalar quantization with a full-precision re-ranking pass over the top 100–200 candidates, and at disk-resident index formats. At this scale the storage architecture is the product decision, and it is worth prototyping two options against your real corpus rather than trusting published benchmarks, which are run on datasets that do not look like yours.

Embedded or edge deployments: a file-backed local index in the application process. This is the right shape for desktop applications, CLI tools, and anything that ships to a customer's machine, and it removes an entire class of network and availability problems.

Two cross-cutting rules apply regardless of which box you land in. First, whatever you pick, keep the chunks and the raw vectors in cheap durable storage so any migration is a bulk reload. Second, prefer the option your team can debug at 2 a.m. — an unfamiliar distributed system with excellent benchmark numbers is worse in production than a familiar one with adequate numbers.

Related questions

Do I need a dedicated vector database at all?

Usually not below about 100,000 vectors. A vector extension on a database you already operate gives transactional consistency, real SQL filtering, and one backup story. Move to a dedicated store when filtered-search performance, quantization, or scale genuinely forces it.

What happens when I change embedding models?

Every stored vector becomes incompatible with new query vectors — dimensionality and coordinate space both change. You re-embed the entire corpus. Build the new index alongside the live one, verify recall, then cut over. This is why keeping raw chunk text cheaply readable matters.

Should I store the chunk text in the vector store or separately?

Store an identifier and the minimum metadata needed for filtering in the vector store, and keep authoritative chunk text in your primary datastore or object storage. Many teams duplicate the text into the index for single-hop retrieval, which is fine — just keep the durable copy elsewhere.

How much does quantization actually hurt retrieval quality?

Scalar int8 quantization typically costs a small amount of recall for roughly a 4× memory reduction. Binary quantization is far more aggressive and needs a full-precision re-ranking pass over the top candidates. Measure recall on your own corpus rather than assuming published figures transfer.

Is hybrid search worth the extra complexity?

Yes when your corpus contains identifiers, error codes, product names, or acronyms — dense vectors handle those poorly. Fusing lexical and semantic rankings reliably outperforms either alone. Most mature stores support it natively, so the complexity cost is often just a config change.

FAQ

How do I decide chunk size before I build the index?

Start at 512 tokens with 10–20% overlap and measure. Build a small evaluation set of real questions with known correct source passages, then test 256, 512, and 1024 against it. Smaller chunks improve precision and hurt context; larger chunks do the reverse. This is corpus-dependent enough that a single afternoon of measurement beats any general recommendation, and the answer changes if you add a re-ranking step, which lets you retrieve smaller chunks and expand to neighbors afterward.

What index type should I default to?

HNSW, unless memory is the binding constraint. It gives high recall at low latency, supports incremental inserts without retraining, and its tuning knobs are well understood. Use m around 16–32 and ef_construction around 200 as a starting point, then tune ef_search at query time against measured recall. Choose IVF-family or quantized indexes when the memory footprint of HNSW is genuinely unaffordable, and flat exact search below about 100,000 vectors.

How do I handle access control in retrieval?

Enforce it server-side as a mandatory filter or through physical separation — separate collections or namespaces per tenant. Never let the application layer decide whether to include the ACL filter, because eventually a code path will forget. Store the permission identifier as first-class indexed metadata on every chunk at ingestion time, and test the narrow-filter case explicitly: a filter matching a tiny fraction of the corpus is where both correctness and latency problems surface.

Should I self-host the embedding model or use an API?

Volume decides. Intermittent or bursty ingestion favors a hosted API — embedding calls are among the cheapest in the LLM stack, and you avoid operating a GPU. Continuous high-volume ingestion, strict data-residency requirements, or the need to guarantee that the model never changes underneath you all favor self-hosting an open-weight model. A pinned self-hosted model also removes the risk of a provider silently updating a model version and invalidating your index.

How often should I rebuild the index?

Rebuild when recall drifts, when accumulated soft-deleted tombstones inflate memory, or when you change models or chunking. For a stable corpus with light churn, that may be never. For a corpus with heavy update rates, schedule a periodic compaction or rebuild and monitor index size against live vector count — a growing gap between them is the signal. Rehearse the rebuild before you need it, so it is a routine operation rather than an incident.

What should I monitor in production?

Four things: p50 and p99 retrieval latency split by filtered versus unfiltered queries, recall@k against a fixed ground-truth query set run on a schedule, the ratio of index size to live vector count, and the rate of queries returning fewer results than requested. That last one is the clearest early signal of a filter-path problem, and it is the metric teams most often lack when a retrieval quality complaint arrives.

Sources

• https://arxiv.org/abs/1603.09320 — Malkov & Yashunin, the original HNSW paper
• https://github.com/pgvector/pgvector — pgvector, vector similarity search for PostgreSQL
• https://github.com/facebookresearch/faiss — FAISS, Meta's similarity search library
• https://faiss.ai/ — FAISS documentation, including index type selection guidance
• https://arxiv.org/abs/2005.11401 — Lewis et al., the original retrieval-augmented generation paper
• https://www.postgresql.org/docs/current/indexes.html — PostgreSQL indexing documentation
• https://huggingface.co/spaces/mteb/leaderboard — MTEB, the massive text embedding benchmark leaderboard
• https://arxiv.org/abs/2205.13147 — Matryoshka Representation Learning, the basis for dimension truncation
• https://github.com/nmslib/hnswlib — hnswlib, a header-only HNSW implementation with tuning parameters documented

Related on PULSE

• How do you chunk documents for retrieval-augmented generation?
• What is hybrid search and when does it beat pure vector search?
• How do you evaluate RAG retrieval quality?
• When should you re-rank retrieved results?
• How do you migrate to a new embedding model without downtime?
• What does a production RAG ingestion pipeline look like?