How do you build a vector database from scratch for RAG in 2027?

The Two Main Paths: Dedicated Engines vs. Custom Implementations

When you decide to build a vector database from scratch for RAG in 2027, the first fork in the road is whether "from scratch" means configuring a purpose-built vector database or literally writing the indexing and search code yourself. Both interpretations are valid, but they lead to dramatically different projects.

The dedicated engine path is what most production RAG systems actually use. You take something like pgvector (a PostgreSQL extension), Qdrant, Weaviate, Milvus, or Chroma, and you build your database schema, ingestion pipeline, and query layer around it. The phrase "from scratch" here means you are not using a managed cloud service — you are provisioning the database yourself, designing the schema, and writing the integration code. This path gives you control over infrastructure costs, data residency, and customization, while offloading the hard parts of approximate nearest neighbor (ANN) search to a battle-tested engine.

The truly custom path means implementing vector storage and search yourself. You might use FAISS or hnswlib as a library (these are not databases, just index structures), and then build persistence, metadata filtering, concurrency control, and a query API around them. This is a legitimate approach for learning, for highly specialized use cases, or for environments where you cannot run a separate database service. In 2027, you might also consider SQLite with the sqlite-vec extension, which gives you a file-based vector database that is genuinely built from scratch in the sense that you assemble the pieces yourself.

The trade-offs between these paths are substantial. A dedicated engine gives you transactional guarantees, backup and restore tooling, role-based access control, and a query language. A custom implementation gives you total control over memory layout, index parameters, and the ability to optimize for your exact workload. Most teams should start with a dedicated engine and only go fully custom when they have measured a specific bottleneck that the engine cannot solve.

How to Decide Between the Paths

The decision between using a dedicated vector database engine and building a custom index layer depends on your team's constraints, your data volume, and your latency requirements. A small team with a prototype and a tight deadline should not be writing ANN search from scratch. A team with millions of vectors, sub-50-millisecond latency requirements, and a need to optimize index parameters at the hardware level might need the custom path.

The key insight is that your choice of vector database is not permanent. You can start with a simple embedded option like Chroma or sqlite-vec, prove your RAG pipeline works, and then migrate to a distributed engine as your corpus grows. The migration path is well-worn because most teams follow exactly this trajectory. What matters is that you design your ingestion and query code against a stable interface so that swapping the underlying database is a configuration change, not a rewrite.

For a RAG system in 2027, the practical decision criteria are: team size (a two-person team should not operate a Milvus cluster), data growth rate (if you add millions of vectors per month, plan for distributed early), and query latency requirements (sub-100ms is achievable with most engines, but sub-10ms at scale requires careful index tuning). Also consider your existing infrastructure — if you already run PostgreSQL, pgvector is the lowest-friction addition. If you are on Kubernetes and comfortable operating stateful services, Qdrant or Milvus are strong choices.

Concrete Numbers Behind Each Option

Understanding the concrete performance and resource characteristics of each vector database option helps ground your decision. These numbers are typical ranges observed in production systems, not guarantees, but they give you a sense of what to expect.

pgvector with HNSW indexing on a single PostgreSQL instance handles up to about 5 million vectors of 1536 dimensions (the output size of OpenAI's text-embedding-3-large) with sub-100ms query times on modest hardware — say 8 vCPUs and 32GB RAM. The index size for 1 million 1536-dimensional vectors is roughly 2-3GB, depending on HNSW parameters. Insert throughput is around 500-2000 vectors per second with HNSW index maintenance, which is fine for batch ingestion but slow for real-time streaming.

Qdrant on a single node with 16GB RAM handles roughly 10-50 million vectors of the same dimensionality, with query latency between 5-50ms depending on the number of HNSW neighbors searched. It supports payload indexing for metadata filters, which is critical for RAG systems that filter by document source, date, or access control level. Qdrant's memory-mapped storage means you can exceed RAM capacity, but query latency degrades as you hit disk.

Milvus with a distributed deployment — one coordinator, three query nodes, three data nodes — handles hundreds of millions to billions of vectors. The cost is operational complexity: you are running a distributed system with ZooKeeper or etcd for coordination, object storage for data persistence, and multiple query replicas. A minimal production Milvus cluster costs roughly $500-2000 per month in cloud infrastructure. Query latency stays under 50ms at the 95th percentile with proper index tuning, but you need to monitor memory pressure and index building times carefully.

FAISS with an IVF-PQ index on a single GPU can search 100 million vectors in under 10ms, but you must build the index in batches and handle memory carefully. A 100 million vector corpus of 1536-dimension embeddings requires about 600GB of RAM for a flat index, or about 60GB with product quantization at the cost of some recall. Building such an index takes hours on a single A100 GPU. This path is for teams that have exhausted what single-node vector databases can do.

Chroma and LanceDB are embedded options that run in-process. They handle up to roughly 1-10 million vectors on a single machine with acceptable performance, but they lack the network API, authentication, and multi-tenant isolation of server-based engines. They are excellent for prototyping, local development, and edge deployments where you cannot run a separate database process.

For RAG specifically, the embedding dimension matters as much as the vector count. If you use a smaller embedding model — like the 384-dimension all-MiniLM-L6-v2 or the 768-dimension bge-base-en-v1.5 — your storage and memory requirements drop by 50-75% compared to 1536-dimension embeddings. Many teams in 2027 are finding that smaller, cheaper embedding models with good retrieval performance are a better trade-off than the largest models, especially when combined with hybrid search that also uses keyword matching.

Implementation Details and Sequencing

Building your vector database from scratch for RAG follows a sequence that applies whether you choose an embedded engine or a distributed cluster. The order matters because each step depends on the previous one, and getting the embedding and chunking strategy right before you worry about index parameters saves you from re-ingesting your entire corpus.

Start with your chunking strategy. For RAG, the chunk size and overlap determine retrieval quality more than any other factor. A common starting point is 500-1000 tokens per chunk with 10-20% overlap. Smaller chunks of 200-300 tokens improve precision but increase the number of vectors you store and may miss broader context. Larger chunks of 1500-2000 tokens capture more context but dilute the semantic focus, making retrieval less precise. In 2027, many teams are using recursive character text splitting with separators that respect document structure — splitting on headings, paragraphs, and sentences rather than fixed token counts.

Next, select your embedding model. The choice of embedding model determines the vector dimension, the semantic quality, and the cost of ingestion. In 2027, the landscape includes OpenAI's text-embedding-3-large (3072 dimensions with dimensions API), Cohere's embed-v4, Google's Gemini embeddings, and a wide range of open-source models from the MTEB leaderboard. For most RAG systems, a 768-1024 dimension model strikes a good balance between quality and storage cost. If you are building from scratch, you should benchmark at least two or three embedding models on a sample of your actual documents before committing.

Then design your schema. Your vector database needs at least three things: the vector column or collection, metadata columns for filtering, and a primary key that links back to your source documents. In pgvector, this looks like a table with a vector column, a JSONB metadata column, and a text chunk column. In Qdrant, this is a collection with named vectors, payload fields, and point IDs. The metadata you store should include document ID, chunk index, source URL or file path, and any access control tags.

The ingestion pipeline needs to handle failures gracefully. If your embedding API rate-limits you, you need retry logic with exponential backoff. If a document fails to parse, you need to log it and continue. If you are ingesting millions of documents, you need parallel workers — typically 10-50 concurrent embedding requests — and a way to resume from the last successful checkpoint. Most teams build this as a Python or TypeScript service that reads from a message queue, embeds chunks, and writes to the vector database in batches of 100-1000 vectors.

Index building is where the database-specific tuning happens. For HNSW indexes, the key parameters are M (the number of connections per node) and efConstruction (the size of the dynamic candidate list during insertion). Higher M values improve recall but increase memory usage and query time. A common starting point is M=16 and efConstruction=200, with ef (the query-time parameter) set between 50-200 depending on your recall target. For IVF indexes, you choose the number of clusters (nlist) — typically the square root of your vector count — and the number of probes at query time (nprobe), usually 10-50.

After your index is built, you need a query pipeline that goes beyond simple vector search. A production RAG system in 2027 typically does hybrid search: it combines vector similarity with keyword matching (BM25 or similar) and merges the results. It also applies metadata filters before or after the vector search, depending on the engine. Then it reranks the top 20-50 results using a cross-encoder model like Cohere Rerank or a smaller local model, before passing the top 3-10 chunks to the LLM.

The query pipeline also needs to handle the case where the vector search returns poor results. You should log query embeddings, retrieved chunk IDs, and relevance scores so you can analyze failures and tune your chunking, embedding, and index parameters. This observability layer is often neglected but is critical for iterating on RAG quality.

Operational Considerations and Common Pitfalls

Running a vector database in production requires attention to several operational concerns that are easy to overlook when you are building from scratch. The first is backup and recovery. Your vector database is a source of truth for your RAG system — if you lose it, you must re-embed your entire corpus, which can take days. Most engines support snapshots or logical backups, but you need to test restoration procedures, not just assume they work.

The second concern is index maintenance over time. As you add new vectors, HNSW indexes degrade slightly in performance and recall. Some engines rebuild indexes automatically, others require manual optimization. A common pattern is to batch inserts and trigger an index optimization after each batch of 100,000-1,000,000 vectors. If you are continuously ingesting new documents, you may need to schedule index rebuilds during low-traffic windows.

The third pitfall is metadata filter interaction with vector search. Many engines apply metadata filters after the ANN search, which means your filter can exclude all the nearest neighbors and leave you with poor results. The solution is to use engines that support pre-filtering or to design your metadata so that filters are selective enough to work well with post-filtering. You should benchmark query latency and recall with your actual filter patterns, not just with unfiltered searches.

A fourth issue is embedding model drift. If you update your embedding model, the vectors it produces are not comparable to vectors from the old model. This means you cannot mix old and new embeddings in the same index. You need to either re-embed your entire corpus or maintain separate collections for different embedding versions and query both at runtime. This is a strong argument for choosing an embedding model early and sticking with it unless you have a compelling reason to change.

The fifth pitfall is cost estimation. Storing vectors is cheap — a few dollars per million vectors per month in object storage. Embedding generation is more expensive, especially if you use paid APIs. The compute cost for embedding 1 million chunks with a 1024-dimension model is roughly $10-50 depending on the provider. The bigger cost is often the LLM calls for generation, not the vector database itself. But if you are running a self-hosted vector database on cloud VMs, the infrastructure cost is a fixed monthly expense that you should budget for.

Related Questions

How do you choose an embedding model for RAG?

Benchmark 2-3 models on a sample of your documents using retrieval metrics like recall@10 and nDCG. Consider dimension size, latency, cost per token, and whether the model supports the languages in your corpus. Smaller models often perform within 5-10% of larger ones at a fraction of the cost.

What is the best chunk size for RAG retrieval?

The optimal chunk size depends on your document type and query patterns. Start with 500-1000 tokens with 10-20% overlap. Test smaller chunks if you need precise answers, larger chunks if your queries require broader context. Use recursive splitting that respects document structure.

How do you evaluate RAG retrieval quality?

Create a test set of 50-100 queries with known relevant chunks. Measure recall@k, precision@k, and mean reciprocal rank. Compare your hybrid search against pure vector search. Track failure cases where the correct chunk is not in the top 10 results.

Can you use a relational database for vector search?

Yes. PostgreSQL with pgvector supports exact and approximate nearest neighbor search. It is a solid choice for teams already running PostgreSQL, with the caveat that it handles fewer vectors than dedicated engines before performance degrades.

How do you handle updates and deletions in a vector database?

Most vector databases support point updates and deletions, but the index may not immediately reflect changes. Some engines use tombstones and rebuild segments. Plan for a delay between write and search visibility, and design your RAG system to tolerate this.

FAQ

What is the difference between a vector database and a vector index library?

A vector database is a complete data management system that handles storage, indexing, querying, and typically offers features like transactions, backup, and access control. A vector index library like FAISS or hnswlib provides only the indexing and search algorithms. You must build persistence, concurrency, and a query API yourself when using a library.

How much does it cost to run a vector database for RAG?

For a small system with under 1 million vectors, you can run pgvector on a $50-100 per month VM. A mid-size system with 10-50 million vectors on Qdrant or Milvus costs $200-1000 per month in infrastructure. The embedding generation cost is separate and depends on your corpus size and chosen model.

What is HNSW and why is it the default index algorithm?

HNSW stands for Hierarchical Navigable Small World, a graph-based algorithm for approximate nearest neighbor search. It offers a good balance of query speed, recall, and memory usage. It builds a multi-layer graph where higher layers have fewer nodes, allowing searches to start coarse and refine at lower layers.

How do you handle multi-tenancy in a vector database?

Use metadata fields for tenant ID and always include tenant ID in query filters. Ensure your index supports pre-filtering on this field. Alternatively, use separate collections or databases per tenant if your engine supports it. Consider data isolation requirements when choosing between these approaches.

What is hybrid search and why does RAG need it?

Hybrid search combines vector similarity with keyword matching. Vector search captures semantic meaning but can miss exact terms, IDs, or acronyms. Keyword search handles these cases but misses synonyms and paraphrases. Merging both result sets improves retrieval quality, especially for technical or domain-specific queries.

How do you monitor a vector database in production?

Track query latency percentiles, recall at fixed k, index size, memory usage, and ingestion throughput. Set up alerts for index building failures, high disk usage, and query latency degradation. Log all queries with their retrieved chunks to enable offline analysis of retrieval quality.

Sources

https://www.pgvector.org/ https://qdrant.tech/documentation/ https://milvus.io/docs https://github.com/facebookresearch/faiss https://github.com/nmslib/hnswlib https://www.sqlite.org/forum/ https://python.langchain.com/docs/concepts/vectorstores/ https://huggingface.co/spaces/mteb/leaderboard https://weaviate.io/developers/weaviate https://docs.trychroma.com/

Related on PULSE

• Chunking strategies for RAG: fixed-size vs. semantic splitting
• Embedding model selection: open-source vs. proprietary APIs
• Hybrid search techniques: combining BM25 with vector similarity
• Reranking approaches for improving RAG retrieval accuracy
• Evaluating RAG pipelines: offline metrics vs. online user feedback
• Scaling RAG systems from prototype to production