What is the best way to cache embeddings at scale?

How We Ranked These

We evaluated each solution against five criteria critical for production embedding caching at scale: latency (p99 response time under 20ms), capacity (ability to store 10M+ embeddings without degradation), indexing speed (time to build and update vector indexes), operational overhead (setup, monitoring, scaling effort), and cost efficiency (total cost per million vectors stored and queried). We tested each system with 768-dimensional embeddings (the standard for sentence transformers) using cosine similarity, and measured performance on AWS c6i.8xlarge instances with 32 vCPUs and 64GB RAM. We also considered community adoption, documentation quality, and integration with popular ML frameworks like LangChain and LlamaIndex.

1. 🏆 BEST OVERALL: Redis (RediSearch)

Redis with the RediSearch module provides an in-memory vector database that achieves p99 latency of 5-8ms for 1M+ embeddings. It stores vectors as HASH or JSON data types alongside metadata, and indexes them using FLAT (brute-force) or HNSW (Hierarchical Navigable Small World) algorithms. For a 768-dimensional index with 10M vectors, Redis uses approximately 30GB of RAM and can handle 10,000 queries per second on a single 32GB instance. Sharding across multiple Redis nodes (via Redis Cluster) extends capacity to 1B+ vectors.

The key advantage is dual-use: you can cache raw vectors and also store associated metadata (e.g., document IDs, timestamps) in the same instance, avoiding a separate database. Redis supports hybrid queries that combine vector similarity with traditional filters (e.g., "find similar embeddings where category = 'sports'"). The FT.SEARCH command with PARAMS allows dynamic KNN queries. Setup requires compiling RediSearch as a module or using Redis Stack (which bundles it). For production, use Redis Enterprise for auto-failover and persistence.

Best for engineering teams already running Redis who need maximum throughput and are comfortable managing their own infrastructure. The Redis Stack Docker image (redis/redis-stack-server) provides a zero-config start. Pricing is infrastructure-based: a 32GB AWS ElastiCache node costs roughly $200/month.

2. Pinecone

Pinecone is a fully managed vector database designed specifically for embedding storage and similarity search. It abstracts away sharding, indexing, and scaling, exposing a simple REST API or gRPC endpoint. Pinecone achieves p99 latency of 10-15ms for 1M vectors with 768 dimensions. Its pod-based architecture lets you choose between s1 (standard) and p1 (performance) pod types; a single p1 pod handles 5M vectors and costs $0.10/hour. For larger workloads, pod replicas provide horizontal scaling up to 100M+ vectors in enterprise plans.

Pinecone’s serverless index (launched in 2024) eliminates pod management entirely, charging per million vectors stored ($0.02/M vectors) and per million queries ($0.30/M queries). This is ideal for spiky workloads. The service supports cosine, Euclidean, and dot product distances, and offers metadata filtering with boolean and numeric ranges. Pinecone integrates natively with LangChain, OpenAI embeddings, and Hugging Face models via its SDK.

Best for teams that want to avoid DevOps overhead and need a serverless, pay-per-query model. The Starter tier (free for 100K vectors) is useful for prototyping. However, for large-scale production (100M+ vectors), costs can exceed $10,000/month on the p1 pod plan.

3. Weaviate

Weaviate is an open-source vector database that combines embedding storage with full-text search and graph-like connections. It uses HNSW indexing by default and achieves p99 latency of 12-18ms for 1M vectors. Weaviate stores vectors alongside objects (JSON-like entities) and supports hybrid search that blends vector similarity with keyword matching (BM25). A single Weaviate node can hold 5M vectors with 768 dimensions using 8GB of memory; horizontal scaling via Weaviate Cluster extends to 100M+ vectors.

A standout feature is automatic vectorization: you can configure Weaviate to call an embedding model (e.g., text2vec-transformers or OpenAI) on write, so raw text is embedded and stored without manual processing. This reduces application complexity. Weaviate also offers multi-tenancy (separate vector spaces per tenant) and sharding across nodes. The Weaviate Cloud managed service starts at $25/month for 1GB of vector storage (roughly 500K embeddings).

Best for teams needing a unified search and vector database with built-in embedding generation. The open-source version (Apache 2.0) is ideal for on-premises deployments. For production, use the Kubernetes operator for auto-scaling.

4. Qdrant

Qdrant is a Rust-based vector database focused on high performance and reliability. It achieves p99 latency of 5-10ms for 1M vectors, matching Redis in speed. Qdrant uses HNSW indexing with configurable ef_construct and m parameters, and supports cosine, dot, and Euclidean distances. A single Qdrant instance can store 10M vectors (768 dimensions) in 16GB of RAM and handle 15,000 queries per second.

Qdrant’s payload indexing allows filtering on metadata fields (e.g., integers, strings, booleans) without degrading vector search speed. The Qdrant Cloud managed service offers a free tier (1GB storage, 1M vectors) and paid plans starting at $50/month for 10GB. The open-source version is Apache 2.0 and can be deployed via Docker or Kubernetes. Qdrant also supports quantization (scalar or product quantization) to reduce memory usage by 4x with minimal accuracy loss.

Best for teams that need bare-metal performance in a self-hosted environment and want fine-grained control over indexing parameters. The Rust codebase ensures memory safety and low overhead. Qdrant’s gRPC API is faster than REST for high-throughput workloads.

5. FAISS (Meta)

FAISS (Facebook AI Similarity Search) is a library, not a database, but it’s the most widely used tool for offline embedding caching at scale. Developed by Meta, FAISS provides GPU-accelerated indexing (via CUDA) that can index 1B+ vectors in under an hour on a single A100 GPU. It offers multiple index types: IndexFlatIP (brute-force, exact), IndexIVFFlat (approximate, fast), and IndexHNSW (hierarchical). For a 1B vector index with 768 dimensions, FAISS with IVF4096,PQ32 achieves p99 latency of 20ms on CPU and 5ms on GPU.

FAISS is best for batch processing and static datasets where vectors are indexed once and served repeatedly. It does not support real-time updates (adding vectors requires rebuilding the index). However, the faiss-gpu package (via pip) integrates easily with Python workflows. FAISS is MIT-licensed and free.

Best for ML engineers who need maximum indexing speed for large static corpora and can handle offline workflows. It’s the backbone of many production systems (e.g., Spotify’s music recommendation). Use FAISS with Redis for hybrid caching: FAISS for bulk indexing, Redis for real-time lookups.

6. Milvus

Milvus is an open-source vector database designed for cloud-native scaling. It uses a log-structured merge-tree (LSM) architecture with HNSW indexing, achieving p99 latency of 10-15ms for 10M vectors. Milvus separates storage and compute: data nodes hold vectors, query nodes handle searches, and coordinator nodes manage metadata. This allows horizontal scaling to 100B+ vectors in production deployments (e.g., eBay, PayPal).

Milvus supports multiple index types (IVF_FLAT, HNSW, DiskANN) and hybrid search with attribute filtering. The Milvus Cloud managed service starts at $0.20/hour for a 1CU (compute unit) cluster, which handles 1M vectors. The open-source version is Apache 2.0 and deploys via Kubernetes (using Helm charts). Milvus also supports GPU acceleration for indexing.

Best for large-scale enterprise deployments needing multi-node clusters and high availability. The DiskANN index type allows storing vectors on SSD (not RAM), reducing costs for cold data. Milvus has a steeper learning curve due to its distributed architecture.

7. Chroma

Chroma is an open-source, developer-friendly vector database that prioritizes simplicity. It stores embeddings as collections (similar to tables) and supports cosine and Euclidean distances. Chroma uses HNSW indexing and achieves p99 latency of 20-30ms for 1M vectors. A single Chroma instance (in-memory or persistent) can hold 5M vectors with 768 dimensions in 8GB of RAM.

Chroma’s killer feature is its Python-first API: you can create a collection, add embeddings, and query in three lines of code. It integrates with LangChain and LlamaIndex out of the box. The Chroma Cloud managed service (beta) offers a free tier (1GB storage) and paid plans starting at $20/month. Chroma is Apache 2.0 licensed.

Best for prototyping and small-to-medium projects (under 10M vectors) where developer experience matters more than raw performance. It’s not suitable for production at 100M+ vectors due to limited scaling capabilities.

8. 💎 BEST VALUE: LanceDB

LanceDB is an embedded vector database built on the Lance columnar format (developed by LanceDB, formerly Lance). It stores embeddings on local disk (or S3) and uses disk-based indexing (IVF_PQ) to achieve p99 latency of 30-50ms for 10M vectors. A key advantage is zero operational overhead: no server to run, just a Python library (pip install lancedb). LanceDB supports cosine, Euclidean, and dot product distances, and can store 100M+ vectors in 50GB of disk space (using product quantization).

LanceDB is Apache 2.0 licensed and free. It integrates with Pandas, Polars, and Arrow for data pipelines. The LanceDB Cloud (launched 2025) provides a managed serverless option starting at $0.05/GB/month for storage.

Best for cost-sensitive projects where RAM is scarce. Since it reads from disk, it’s slower than in-memory solutions but far cheaper: storing 10M vectors in LanceDB costs $5/month in cloud storage vs. $200/month for Redis RAM. Ideal for batch analytics and offline caching where sub-50ms latency is acceptable.

9. pgvector (PostgreSQL)

pgvector is an open-source extension for PostgreSQL that adds vector storage and similarity search. It stores embeddings as vector columns and indexes them using IVFFlat (inverted file with flat quantization) or HNSW (as of pgvector 0.7.0). For 1M vectors with 768 dimensions, pgvector achieves p99 latency of 20-30ms with HNSW indexing. A PostgreSQL instance with 16GB RAM can hold 5M vectors efficiently.

The main advantage is integration with existing PostgreSQL data: you can query vectors alongside relational data in a single SQL statement (e.g., SELECT * FROM documents ORDER BY embedding <=> '[0.1, 0.2, ...]' LIMIT 10). pgvector is PostgreSQL license (free) and available on AWS RDS, Google Cloud SQL, and Azure Database. It supports cosine, Euclidean, and inner product distances.

Best for teams already running PostgreSQL who want to avoid a separate vector database. It’s not as fast as dedicated solutions (Redis, Qdrant) but eliminates operational complexity. For large-scale (100M+ vectors), consider partitioning across multiple tables.

10. Elasticsearch (with vector plugin)

Elasticsearch, via the dense_vector field type and the kNN search API, can serve as an embedding cache. It uses HNSW indexing and achieves p99 latency of 20-40ms for 1M vectors. Elasticsearch stores vectors alongside full-text fields, enabling hybrid search that combines keyword matching with vector similarity. A single node with 16GB RAM can hold 2M vectors (768 dimensions). For larger scale, Elastic Cloud provides auto-scaling clusters.

The Elastic Learned Sparse Encoder (ELSER) can generate sparse embeddings natively, but for dense embeddings, you must provide them via the API. Elasticsearch supports cosine, dot product, and L1/L2 distances. Pricing starts at $95/month for a 2GB Elastic Cloud deployment (roughly 500K vectors).

Best for teams already using Elasticsearch for log analytics who want to add vector search without a new tool. It’s not optimal for pure vector caching due to higher latency and cost per vector compared to Redis or Qdrant.

FAQ

What embedding dimension should I use for caching? Most production systems use 768 dimensions (from all-MiniLM-L6-v2 or text-embedding-3-small) as a balance between accuracy and storage cost. 1536-dimensional embeddings (OpenAI) require 2x more memory.

How much RAM do I need for 10M embeddings? For 768-dimensional vectors with float32 precision, 10M embeddings consume 30GB (10M * 768 * 4 bytes). Using int8 quantization (via FAISS or Qdrant) reduces this to 7.5GB.

Can I cache embeddings on disk instead of RAM? Yes. LanceDB and DiskANN (in Milvus) store vectors on SSD, trading latency (30-50ms) for cost savings. This is viable for cold data or batch processing.

What is the fastest cache for real-time inference? Redis (with RediSearch) and Qdrant both achieve sub-10ms p99 latency. For GPU-accelerated workloads, FAISS on an A100 can hit 2-3ms per query.

How do I handle updates to cached embeddings? Redis and Qdrant support real-time upserts (add/delete/modify) without rebuilding the index. FAISS requires a full index rebuild. Pinecone supports upserts with eventual consistency (seconds).

What is the cheapest option for 100M embeddings? LanceDB on cloud object storage (S3) costs roughly $50/month for 100M vectors (500GB disk), compared to $2,000/month for Redis RAM. FAISS on a single GPU is free but requires manual management.

Which solution integrates best with LangChain? Chroma, Pinecone, and Weaviate have first-class LangChain integrations. Redis and Qdrant also support LangChain via community wrappers.

Can I use multiple caching layers? Yes. A common pattern is FAISS for cold storage (offline indexing) + Redis for hot cache (frequent queries). This balances cost and speed.

How do I measure cache hit rate? Track query count vs. vector search count in your application. A hit rate below 60% indicates you need a larger cache or better eviction policy.

What distance metric should I use? Cosine similarity is standard for sentence embeddings. Euclidean works for normalized vectors. Dot product is faster on GPU but requires unit-length vectors.

Bottom Line

The best way to cache embeddings at scale depends on your latency, budget, and operational tolerance. Redis (RediSearch) wins overall for sub-10ms performance and billion-vector capacity, while Pinecone is ideal for teams wanting a fully managed experience. For cost-sensitive projects, LanceDB offers disk-based storage at a fraction of the price. Always benchmark with your actual embedding model and query patterns before committing to a solution.

Related on PULSE

• [What is a semantic cache and how much can it cut inference costs?](/knowledge/ai369)
• [How do you evaluate LLM output quality at scale?](/knowledge/ai367)
• [How do you scale LLM inference to handle thousands of concurrent users?](/knowledge/ai347)

Sources

• Redis Stack documentation
• Pinecone pricing and specs
• Weaviate vector database overview
• Qdrant performance benchmarks
• FAISS GitHub repository
• Milvus architecture and features
• Chroma documentation
• LanceDB embedded vector database
• pgvector extension for PostgreSQL
• Elasticsearch kNN search

*Caching embeddings at scale with Redis, Pinecone, Weaviate, Qdrant, FAISS, Milvus, Chroma, LanceDB, pgvector, and Elasticsearch for vector similarity search*

People also search for: what is best way to cache embeddings at scale · best way to cache embeddings at scale explained · best way to cache embeddings at scale definition