What infrastructure does retrieval-augmented generation require?

What it is and why it matters

Retrieval-augmented generation (RAG) is an architecture that combines information retrieval with text generation, allowing large language models to answer questions using external knowledge rather than relying solely on their training data. The infrastructure for RAG determines query latency, accuracy, scalability, and cost—factors that directly impact user experience and operational viability. A poorly designed RAG infrastructure can introduce retrieval latencies exceeding 500 milliseconds, hallucination rates above 15%, and costs that spiral to $0.10 per query or more. Conversely, a well-architected stack can achieve p95 latencies under 200 milliseconds, retrieval precision above 90%, and per-query costs below $0.005. The infrastructure choices also affect data governance, compliance with regulations like GDPR and HIPAA, and the ability to handle revenue-critical applications where incorrect answers could lead to financial losses. Understanding the full infrastructure stack—from document ingestion pipelines to vector databases, embedding services, LLM inference endpoints, and orchestration frameworks—is essential for any team building production RAG systems.

The core components of RAG infrastructure include the ingestion pipeline for processing source documents, the embedding service for converting text into vector representations, the vector database for storing and retrieving those embeddings, the orchestration layer that coordinates retrieval and generation, and the LLM inference endpoint that produces final answers. Each component introduces specific latency, cost, and accuracy trade-offs that must be balanced against application requirements. For example, using a dense embedding model like text-embedding-3-small (1536 dimensions) provides strong semantic understanding but costs $0.13 per million tokens to run via API, while a self-hosted model like all-MiniLM-L6-v2 (384 dimensions) eliminates per-token costs but requires GPU infrastructure and delivers lower retrieval precision. Similarly, a managed vector database like Pinecone offers sub-10ms retrieval latency with zero operational overhead but costs $0.10 per million vectors per hour, while a self-hosted solution like Milvus requires significant DevOps investment but can reduce costs by 60-80% at enterprise scale.

The orchestration layer is often the most underestimated component. LangChain and LlamaIndex provide the plumbing to connect retrieval and generation, but they introduce their own latency overhead (typically 50-150ms per query for prompt assembly and context management). Teams must also consider the prompt template design—how retrieved chunks are formatted, how many chunks to include, and how to handle cases where no relevant chunks are found. A robust orchestration layer includes fallback logic for low-confidence retrievals, reranking to improve chunk ordering, and dynamic chunk selection based on token budget. Without this infrastructure, RAG systems produce inconsistent answers and waste tokens on irrelevant context.

Security infrastructure is another critical but often overlooked component. RAG systems expose two attack surfaces: the retrieval pipeline (where prompt injection can extract proprietary document content) and the generation pipeline (where malicious queries can bypass safety filters). Production infrastructure must include input sanitization, output validation, rate limiting, and audit logging. For revenue-critical applications handling financial data or customer information, teams should also implement data encryption at rest and in transit, access controls for document collections, and compliance monitoring for regulations like GDPR and HIPAA.

The step-by-step process

Building a RAG infrastructure follows a defined pipeline that transforms raw documents into grounded answers. The process begins with document ingestion, where source files (PDFs, HTML, databases, Confluence pages) are loaded, parsed, and normalized. Documents are then chunked into segments of 256–1024 tokens using recursive character text splitting or semantic splitting, with an overlap of 10–20% to preserve context across boundaries. Each chunk is passed through an embedding model—such as OpenAI's text-embedding-3-small (1536 dimensions) or a self-hosted model like all-MiniLM-L6-v2 (384 dimensions)—which converts the text into a dense vector representation. These vectors are indexed in a vector database (Pinecone, Weaviate, Qdrant, Milvus, pgvector, or Chroma) along with metadata for filtering.

On query execution, the user's question is embedded using the same model, then a vector search retrieves the top-k most similar chunks (typically k=3–10). The retrieved chunks are assembled into a prompt with the original question, and this context is sent to an LLM (GPT-4o, Claude 3.5, Llama 3.1) for answer generation. The generated response is optionally post-processed for formatting, citation extraction, and safety filtering before delivery. The chunking strategy directly impacts retrieval quality: chunks that are too small (under 256 tokens) lose context and reduce precision by 20-30%, while chunks that are too large (over 1024 tokens) dilute relevance and increase token costs by 2-4x. Semantic splitting, which uses document structure (headings, paragraphs, code blocks) rather than fixed token counts, typically improves retrieval precision by 15-25% compared to naive character-based splitting.

The embedding generation step is a significant bottleneck in the pipeline. A single NVIDIA T4 GPU can embed approximately 2,000 documents per minute using text-embedding-3-small, while CPU-based embedding achieves only 200-400 documents per minute. For ingestion pipelines processing 1 million documents, GPU acceleration reduces embedding time from 42 hours to just 8 hours. The vector database index building also requires careful tuning: HNSW (Hierarchical Navigable Small World) graphs with ef_construction=200 and M=16 provide excellent search speed but consume 2-3x more memory than the raw vectors. For datasets exceeding 10 million vectors, DiskANN or IVF_FLAT with product quantization reduces memory usage by 4-8x while maintaining 90%+ recall.

The orchestration layer manages the flow between retrieval and generation, handling prompt construction, context window management, and response validation. LangChain's LCEL (LangChain Expression Language) provides a declarative way to compose RAG chains, while LlamaIndex offers more sophisticated node parsing and index management. Both frameworks support dynamic chunk selection based on relevance scores, reranking using cross-encoders (like Cohere rerank or BAAI/bge-reranker-v2-m3), and fallback strategies for low-confidence retrievals. Production systems typically add a caching layer (Redis or in-memory) that stores query-answer pairs for identical questions, achieving 10-50ms response times versus 200-500ms for full RAG. Cache hit rates of 20-40% are typical for production systems, reducing average latency by 30-50% and cutting API costs by 15-30%.

Costs, timelines, and typical ranges

The cost of RAG infrastructure varies dramatically based on scale, latency requirements, and deployment model. For a proof-of-concept handling 10,000 documents and 1,000 queries per month, costs are minimal: a laptop with 16GB RAM running Chroma and a quantized Llama 3.1 8B model via llama.cpp costs nothing beyond hardware. At small production scale (100,000 documents, 100,000 queries per month), a managed stack with Pinecone serverless ($0.10 per million vectors per hour, roughly $30/month for 100K vectors), OpenAI text-embedding-3-small ($0.13 per million tokens, ~$5/month), and GPT-4o ($2.50 per million input tokens, ~$25/month) totals approximately $60–$80 per month plus compute for orchestration.

At medium production scale (1 million documents, 1 million queries per month), costs rise significantly: Pinecone serverless at $200–$400/month, embedding API calls at $50–$100/month, and GPT-4o generation at $250–$500/month, totaling $500–$1,000 per month. At enterprise scale (10 million documents, 10 million queries per month), self-hosted infrastructure becomes cost-effective: a 3-node Milvus cluster on AWS r6i.2xlarge instances ($0.40/hour each, ~$900/month), self-hosted embedding with a T4 GPU ($0.35/hour, ~$250/month), and self-hosted Llama 3.1 70B on 4x A100 GPUs ($4.00/hour, ~$2,900/month) totals approximately $4,000–$5,000 per month. Timeline for a production RAG system ranges from 2–4 weeks for a managed stack with existing data, to 3–6 months for a self-hosted enterprise deployment requiring custom embedding models, fine-tuned LLMs, and compliance certifications.

The cost breakdown by component reveals where teams should optimize. Embedding generation typically accounts for 10-15% of total costs, vector database storage 20-30%, LLM inference 40-50%, and orchestration/compute 10-20%. For managed services, API costs dominate at small scales (80% of total), while at enterprise scale, compute costs for self-hosted infrastructure become the largest line item (60-70%). The choice of LLM has the most dramatic impact on cost: GPT-4o-mini costs $0.15 per million input tokens versus GPT-4o at $2.50 per million—a 16x difference. Routing simple factual queries to cheaper models and reserving expensive models for complex reasoning can reduce generation costs by 40-60% without sacrificing answer quality.

Storage costs also vary significantly between vector databases. Pinecone serverless charges $0.10 per million vectors per hour, which for 10 million vectors at 1536 dimensions (approximately 6GB of vector data) translates to $720/month. Self-hosted Milvus on AWS r6i.2xlarge instances (8 vCPU, 64GB RAM, $0.40/hour) costs $288/month for a 3-node cluster, a 60% savings. However, self-hosted solutions require DevOps expertise for cluster management, backup, and scaling. For teams without dedicated infrastructure engineers, the 20-30% premium for managed services is often justified by reduced operational overhead.

Where teams get it wrong

The most common failure in RAG infrastructure is underestimating the importance of chunking strategy. Teams often use default chunk sizes (500 tokens with 50 token overlap) without considering document structure, leading to context fragmentation where a single answer requires information split across multiple chunks. This reduces retrieval precision by 20–40% and increases hallucination rates. Another frequent mistake is treating the vector database as a black box without understanding index parameters: using HNSW with default ef_construction (200) and M (16) works for small datasets but causes memory blowup beyond 1 million vectors, where DiskANN or IVF_FLAT with proper quantization is required. Teams also overlook metadata filtering—without pre-filtering by date, source, or category, vector search returns irrelevant results that confuse the LLM.

On the generation side, the most costly error is using a single LLM endpoint for all queries without routing: simple factual queries ($0.0005 per query with GPT-4o-mini) get routed to expensive models ($0.01 per query with GPT-4o), inflating costs 20x unnecessarily. Security infrastructure is frequently neglected: without input sanitization, prompt injection attacks can extract proprietary document content; without rate limiting, a single user can exhaust the embedding API budget in minutes. Finally, teams often skip monitoring infrastructure, leaving them blind to retrieval drift (where document updates change vector distributions) and generation quality degradation (where LLM updates alter response style).

The orchestration layer introduces its own failure modes. Teams using LangChain or LlamaIndex often accept default prompt templates without testing, resulting in inconsistent answer formatting and citation quality. The default prompt "Answer the question based on the context below" produces different results than "Use the provided context to answer the question. If the context doesn't contain the answer, say 'I cannot find this information.'" without explicit instructions. Similarly, the number of retrieved chunks matters: retrieving 3 chunks works for simple factual questions but fails for multi-hop reasoning that requires synthesizing information across documents. Teams should test retrieval at k=3, 5, 7, and 10 to find the optimal trade-off between context completeness and token cost.

Another common mistake is ignoring the embedding model update cycle. When embedding models are updated (e.g., OpenAI releases text-embedding-3-large), existing vectors become incompatible with new queries. Teams must either re-embed the entire corpus (costing time and money) or maintain multiple embedding indices. Versioning embedding models and tracking which documents use which model version is essential for production systems. Similarly, LLM updates can change response style and accuracy overnight—Claude 3.5 Sonnet v2 produces different answers than v1 for the same prompts. Production infrastructure should pin model versions and test new versions in staging before rolling to production.

Decision framework: when to choose what

Selecting the right RAG infrastructure depends on five key factors: data volume, latency requirements, deployment preference (managed vs. self-hosted), data sovereignty needs, and budget. For teams under 100,000 documents with minimal latency requirements (under 500ms), Chroma or pgvector provide the simplest path—Chroma for rapid prototyping on a single machine, pgvector for teams already using PostgreSQL who want zero additional infrastructure. For 100,000 to 10 million documents requiring sub-100ms latency, managed services dominate: Pinecone for teams prioritizing speed and minimal ops, Weaviate Cloud for organizations needing hybrid search with BM25 keyword scoring. Above 10 million documents, Milvus with GPU-accelerated indexing becomes necessary, though it requires significant DevOps investment.

For regulated industries requiring data sovereignty (finance, healthcare, government), self-hosted Weaviate or Milvus on-premises is mandatory—Pinecone and other cloud-only services cannot be used. For budget-constrained teams, pgvector (free) with a self-hosted embedding model and Llama 3.1 8B provides a functional RAG stack at approximately $200/month for cloud VMs, though throughput is limited to 500 queries per second. The orchestration layer decision is simpler: LangChain for teams needing maximum flexibility to swap components, LlamaIndex for teams building complex multi-hop or agentic RAG workflows. For revenue-critical applications where incorrect answers directly impact revenue, invest in the full stack: managed vector database, high-quality embedding model, routed LLM inference (cheap model for simple queries, expensive model for complex ones), and comprehensive monitoring with LangSmith or similar observability platforms.

The embedding model choice follows a clear hierarchy. For general-purpose retrieval, text-embedding-3-small (1536 dimensions, $0.13/million tokens) offers the best price-performance ratio. For domain-specific retrieval (legal, medical, technical), fine-tuned models like BAAI/bge-large-en-v1.5 (1024 dimensions) or Cohere embed-english-v3.0 (1024 dimensions) improve precision by 5-15%. For multilingual applications, multilingual-e5-large (1024 dimensions) or text-embedding-3-small with multilingual support covers 100+ languages. For budget-constrained teams, all-MiniLM-L6-v2 (384 dimensions, free, self-hosted) provides 85-90% of the retrieval quality of commercial models at zero per-query cost.

The LLM selection should match the complexity of the task. For simple factual queries (product specs, documentation lookups), GPT-4o-mini ($0.15/million input tokens) or Llama 3.1 8B ($0.05/million tokens self-hosted) provides adequate quality. For complex reasoning (multi-hop questions, synthesis across documents, numerical analysis), GPT-4o ($2.50/million input tokens) or Claude 3.5 Sonnet ($3.00/million input tokens) is necessary. For domain-specific generation (legal contracts, medical diagnoses, financial reports), fine-tuned models like Llama 3.1 70B fine-tuned on domain data outperform general models by 20-30% on accuracy metrics. The routing infrastructure to direct queries to the appropriate model based on complexity scores or keyword matching can reduce costs by 40-60%.

Related questions

What is the minimum hardware for running a RAG system?

A prototype with 100,000 documents runs on a laptop with 16GB RAM using Chroma and a quantized Llama 3.1 8B model. Production requires a cloud VM with 32GB RAM and a GPU (NVIDIA T4 or better) for embedding generation.

Which vector database is fastest for retrieval?

Pinecone achieves 8ms p95 latency for 10 million vectors with 1536 dimensions. Milvus with GPU acceleration reaches 12ms for the same dataset. pgvector averages 45ms on a 4-vCPU instance.

Can I run RAG entirely on-premises?

Yes, using Weaviate or Milvus self-hosted, Llama 3.1 70B on your own GPUs, and a local embedding model. This requires at least 4x NVIDIA A100 GPUs and 256GB RAM for production workloads.

How do I handle document chunking for RAG?

Use recursive character text splitting with chunk size 512 tokens and overlap 128 tokens. For code or tables, use semantic splitting based on document structure. LlamaIndex and LangChain both have built-in splitters.

What is the cost of a production RAG system?

For 1 million queries per month with 10 million vectors, expect $500–$2,000 per month: $200 for Pinecone serverless, $150 for GPT-4o generation, $100 for embedding API calls, and $50 for compute.

FAQ

What is the difference between dense and sparse vector retrieval in RAG? Dense vectors (from models like text-embedding-3-small) capture semantic meaning but struggle with exact keyword matches. Sparse vectors (like Elasticsearch's ELSER) excel at term-level matching for domain-specific terms. Hybrid search combining both typically improves recall by 10–20%.

How many chunks should I retrieve per query? For most RAG applications, retrieving 3–10 chunks provides sufficient context without overwhelming the LLM's context window. Retrieving fewer than 3 risks missing relevant information; more than 10 dilutes relevance and increases token costs by 2–5x.

Do I need GPU for embedding generation? For production at scale (100,000+ documents), GPU acceleration reduces embedding time from hours to minutes. A single NVIDIA T4 can embed 1 million documents in approximately 30 minutes. For prototypes under 10,000 documents, CPU-based embedding is acceptable.

Can I use the same embedding model for retrieval and generation? No—embedding models (text-embedding-3-small, all-MiniLM-L6-v2) are specialized for converting text to vectors. Generation requires a separate LLM (GPT-4o, Llama 3.1) trained for text generation. Using an embedding model for generation produces incoherent output.

How do I handle real-time document updates in RAG? Most vector databases support incremental upserts. Pinecone and Weaviate allow adding or updating individual vectors without reindexing the entire corpus. For high-frequency updates (1000+ per second), use Qdrant or Milvus with batch update APIs.

What is the role of caching in RAG infrastructure? Caching frequent queries reduces latency and cost. Redis or in-memory caches store query-answer pairs for identical questions, achieving 10–50ms response times versus 200–500ms for full RAG. Cache hit rates of 20–40% are typical for production systems.

How do I measure RAG infrastructure performance? Track retrieval precision (percentage of retrieved chunks relevant to the query), end-to-end latency (p50, p95, p99), cost per query, and hallucination rate. Use LangSmith or custom monitoring with OpenTelemetry for observability.

Sources

• Pinecone pricing and specifications
• Weaviate documentation on hybrid search
• Chroma GitHub repository
• Qdrant quantization feature documentation
• Milvus index types comparison
• LlamaIndex node parser documentation
• LangChain LCEL guide
• Redis Stack vector search capabilities
• Elasticsearch ELSER overview
• pgvector GitHub repository
• OpenAI text-embedding-3-small pricing
• Llama 3.1 model card on Hugging Face

Related on PULSE

• [How do you secure an LLM application’s infrastructure?](/knowledge/ai363)
• [What infrastructure do you need to run AI agents in production?](/knowledge/ai373)
• [What infrastructure do you need for fine-tuning versus RAG?](/knowledge/ai427)
• [What is the difference between batch and real-time inference infrastructure?](/knowledge/ai409)
• [What is the role of Kubernetes in modern AI infrastructure?](/knowledge/ai431)
• [How do you prevent prompt injection at the infrastructure layer?](/knowledge/ai379)