What is the role of an embedding model in AI infrastructure?

What it is and why it matters

An embedding model is a neural network that takes an input — a paragraph, a support ticket, a product description, a function body — and emits a fixed-length array of floating-point numbers. A common output size is 1024 dimensions; some models emit 384, 768, 1536, 3072, or 4096. The number itself is arbitrary; what matters is that the model was trained so that inputs meaning similar things land near each other in that space, and inputs meaning different things land far apart. "Cancel my subscription" and "how do I end my plan" produce vectors with high cosine similarity even though they share almost no vocabulary. That property is the entire product.

In an AI infrastructure stack, the embedding model sits between raw data and everything downstream. Nothing else in the stack understands your documents. The vector database is a nearest-neighbor index — it stores arrays and answers "which stored arrays are closest to this query array" using HNSW, IVF, or ScaNN. It has no opinion about semantics. The large language model generating the final answer sees only whatever text was retrieved and pasted into its context window. If the embedding model retrieves the wrong five chunks, the LLM answers confidently from the wrong five chunks. Retrieval quality caps generation quality, and the embedding model is retrieval quality.

The practical consequence for teams building on this: embedding choice is a higher-leverage decision than LLM choice for most RAG systems. Swapping from one frontier LLM to another typically moves end-to-end answer accuracy by a few points, because both models reason competently over correct context. Swapping an embedding model that surfaces the right passage in the top 5 for a model that surfaces it at rank 40 changes the answer from correct to fabricated. Teams routinely spend weeks A/B testing generation prompts while running whatever embedding model was in the tutorial they copied.

Four workloads consume embeddings in production. Retrieval for RAG is the loudest: chunk a corpus, embed every chunk, embed the user's question, return the nearest chunks as context. Semantic search is the same machinery exposed directly to users without a generation step. Clustering and deduplication use embeddings to group near-identical support tickets, detect duplicate listings, or collapse redundant documents before indexing — for a company with millions of records, deduplication alone can cut index storage meaningfully. Classification and routing embed an input once, then run a cheap logistic regression or k-nearest-neighbor lookup over labeled examples, which is far faster and cheaper than prompting an LLM per record.

There is a direct revenue line through all of this for commercial deployments. A support deflection bot that retrieves the correct help article deflects a ticket; one that retrieves a plausible-but-wrong article escalates it and burns agent time. A product search that understands "waterproof jacket for hiking" converts; one that keyword-matches "waterproof" and returns tarps does not. The embedding model is the component that decides which of those two systems you shipped, and it is usually the cheapest component in the stack — often a rounding error against inference spend for the generation model.

One more structural fact that surprises teams: embeddings from different models are not interchangeable. A vector produced by model A means nothing in the coordinate space of model B. Changing embedding models means re-embedding your entire corpus and rebuilding the index. That makes the decision semi-permanent and worth benchmarking properly the first time.

The step-by-step process

A production embedding pipeline has two distinct paths that must use the identical model and identical preprocessing: an offline ingestion path that runs once per document (plus on updates), and an online query path that runs on every user request in tens of milliseconds.

Ingestion. Start by extracting clean text from the source format — PDFs, HTML, Confluence exports, transcripts. Extraction quality matters more than most teams expect; a PDF parser that interleaves two-column layouts into scrambled prose poisons every downstream embedding no matter how good the model is. Strip navigation chrome, boilerplate footers, and cookie banners before you embed, or every chunk will carry the same noise and cluster together artificially.

Then chunk. Chunk size is the single most consequential knob teams underestimate. Chunks in the 200–500 token range with 10–20% overlap are a common starting point for question-answering over prose, because they keep one idea per vector and give the retriever a precise target. Larger chunks of 1000+ tokens preserve more context but dilute the vector — the embedding becomes an average of several topics and matches nothing sharply. Respect document structure where you can: split on headings and paragraph boundaries rather than on a fixed character count that cuts sentences in half. Attach metadata to each chunk (source URL, section heading, document date, access-control tags) because filtering by metadata before or during vector search is how you get correctness on top of relevance.

Embed the chunks in batches. Providers accept arrays of inputs per request, and batching is where throughput comes from — embedding one chunk per HTTP call wastes most of the wall-clock time on network round trips. Handle rate limits with exponential backoff, and persist a record of which chunks have already been embedded so a crash halfway through a million-document backfill does not restart from zero.

Upsert into the vector index with a stable ID derived from the source document and chunk position, so re-ingesting an updated document overwrites rather than duplicates. Choose the distance metric your model was trained for — most modern text embedding models are trained for cosine similarity and ship pre-normalized, in which case cosine and dot product are equivalent, but using Euclidean distance on a model trained for cosine quietly degrades results.

Query time. Embed the user's query with the same model. If the model uses asymmetric instruction prefixes — several open-source families distinguish "query:" from "passage:" or ask you to prepend a task instruction — you must apply the prefix consistently at both ingestion and query time. Mismatched prefixes are one of the most common silent failure modes in self-hosted setups. Run approximate nearest-neighbor search for the top 20–50 candidates, apply metadata filters, then optionally rerank with a cross-encoder that scores each candidate against the query directly. Reranking is slower per document but far more accurate than pure vector similarity, so the standard pattern is a wide cheap retrieve followed by a narrow expensive rerank down to the top 3–5 passages actually handed to the LLM.

Costs, timelines, and typical ranges

Embedding is unusually cheap relative to the rest of an AI stack, and teams frequently over-engineer around a cost that turns out to be trivial. Hosted embedding APIs are priced per million input tokens, typically an order of magnitude or two below generation-model pricing for the same token count. The useful mental model: work out your corpus token count, multiply by the per-million rate, and you usually get a one-time backfill cost that is small compared to a single engineer-day.

Sizing a real corpus: a 100,000-document knowledge base averaging 2,000 tokens per document is roughly 200 million tokens. At a rate in the low tens of cents per million tokens, that is a two-figure to low-three-figure dollar backfill — once. Ongoing cost is only new and updated documents plus query embeddings, and query embeddings are tiny because a question is a few dozen tokens. A system serving a million queries a month at 30 tokens per query embeds 30 million tokens monthly, which is negligible.

Storage is where dimension count actually bites. A vector stored as 32-bit floats consumes 4 bytes per dimension. A 1024-dimension vector is about 4 KB; a 3072-dimension vector is about 12 KB. Multiply by chunk count, not document count — a 100,000-document corpus chunked at 400 tokens might produce 500,000 chunks. At 1024 dimensions that is roughly 2 GB of raw vectors; at 3072 dimensions, roughly 6 GB. Add the HNSW graph overhead, which commonly adds another 30–100% on top depending on the connectivity parameter, and managed vector database pricing that scales with stored dimensions, and dimension choice becomes a recurring monthly line item rather than a one-time cost.

This is why dimension-reduction techniques matter. Some models support truncating the vector to a shorter prefix with modest accuracy loss — a property of Matryoshka-style training where the most important information is packed into the leading dimensions. Quantization is the other lever: storing vectors as 8-bit integers instead of 32-bit floats cuts memory roughly 4x, and binary quantization cuts it far more, with the standard mitigation being a rescoring pass over full-precision vectors for the top candidates. On a large index these techniques are the difference between fitting in RAM and paying for disk-backed search with an order-of-magnitude latency penalty.

Latency budgets: a hosted API embedding call for a single short query is typically tens to low hundreds of milliseconds including network. Self-hosted small models on a GPU can return in single-digit milliseconds, which matters when the embedding call sits in the critical path of a search box that must feel instant. ANN search over millions of vectors is typically single-digit to low tens of milliseconds. Cross-encoder reranking of 25–50 candidates adds tens to low hundreds of milliseconds. LLM generation then dominates everything, usually seconds. If your end-to-end feels slow, the embedding model is rarely the culprit — but if you are embedding at query time with a cold self-hosted container, it can be.

Timelines for the work itself: a proof-of-concept RAG pipeline over a few thousand documents is a day or two of engineering. A production pipeline with incremental ingestion, access-control filtering, evaluation harness, and monitoring is typically several weeks. The backfill compute for a large corpus is usually hours, not days, if you batch and parallelize properly — a self-hosted model on a single modern GPU can process hundreds of chunks per second, so a million chunks is measured in hours.

Self-hosting versus API is a crossover calculation. Cloud GPU instances suitable for serving an embedding model rent for low single-digit dollars per hour, which is roughly $1,500–$3,600 a month for one always-on instance before redundancy, monitoring, or engineer time. Below high-volume workloads, the API is simply cheaper and dramatically less operational burden. Self-hosting wins on three axes instead: very high sustained volume, hard data-residency requirements where content cannot leave your network, and latency floors an external API cannot meet. Decide on those axes, not on a spreadsheet comparing per-token rates to zero.

Where teams get it wrong

Benchmarking on the leaderboard instead of on your data. Aggregate benchmark scores average across many datasets that look nothing like your corpus. A model that leads on general web text can badly underperform on clinical notes, legal contracts, or internal ticket shorthand full of product codenames. Build an evaluation set of 100–300 real queries with human-labeled correct documents, then measure recall@5, recall@20, and MRR for each candidate model on your own content. This takes a couple of days and routinely reverses the ranking the leaderboard suggested.

Ignoring chunking entirely. Teams that fix chunking usually get a bigger retrieval improvement than teams that upgrade models. Chunks that are too large produce diffuse vectors; chunks that are too small lose the context needed to be interpretable. A chunk reading "This is not recommended for production use" is useless without the heading it sat under — which is why prepending the document title and section heading to each chunk before embedding is a cheap, high-yield fix.

Mismatched query and document processing. Embedding documents with one model version and queries with another, forgetting instruction prefixes on one side, normalizing one and not the other, or silently upgrading a hosted model version mid-corpus. Every one of these degrades retrieval in ways that look like "the AI is just bad" rather than like a bug. Pin the model version explicitly, log which version produced each vector, and treat a version change as a full re-index.

Expecting semantic search to handle exact matches. Embeddings are bad at part numbers, SKUs, error codes, and rare proper nouns — precisely the tokens users search for most literally. "Error PX-4471" and "Error PX-4417" may embed almost identically. The fix is hybrid search: run BM25 keyword retrieval alongside vector retrieval and fuse the ranked lists, commonly with reciprocal rank fusion. Hybrid consistently beats either method alone on real-world queries, and skipping it is the most common reason a demo that impresses in testing frustrates users in production.

No evaluation harness, so nothing is measurable. Without a labeled query set, every change is a vibe check. Teams tune chunk size, swap models, adjust top-k, and have no idea whether they improved anything. The harness does not need to be sophisticated — a CSV of queries and expected document IDs plus a script that computes recall is enough to make the whole pipeline empirical.

Forgetting access control. Embedding every document into one shared index and retrieving by similarity alone means the model can surface a document the requesting user is not allowed to see. Access-control tags must live in vector metadata and be applied as a pre-filter, not as a post-hoc removal of results — post-filtering can empty your result set and also leaks existence through timing and count.

Treating the index as write-once. Documents change. Without incremental re-embedding on update and deletion on removal, the index slowly fills with stale content that the retriever happily returns as current truth. Deletion in particular gets skipped, and a RAG system confidently citing a policy that was rescinded a year ago is a real business risk.

Decision framework: when to choose what

Work the decision in this order, because each answer constrains the next.

Start with data residency. If contracts or regulation forbid sending content to a third-party API, the decision is made: self-host an open-weight model. Everything downstream is about which open model and what hardware. Do not try to negotiate around this one — it is the only truly hard constraint in the list.

Then language coverage. If a meaningful share of your corpus or your users are non-English, filter to models explicitly trained multilingual and evaluate them on your actual languages. English-optimized models degrade sharply on lower-resource languages, and cross-lingual retrieval — an English query finding a German document — only works if the model was trained to put translations near each other in the same space.

Then modality. Text-only is the default. If you need code search, use a model trained on code, because code and prose have different token distributions and a general text model retrieves poorly over function bodies. If you need image-text search, you need a genuinely multimodal model that places images and captions in one shared space.

Then context length versus chunking. Long-context embedding models let you embed a whole contract as one vector, which is attractive for document-level triage. But a single vector for a 20-page document is a heavy average — it tells you the document is about employment law, not which clause answers the question. For question-answering, chunking plus a normal-context model usually beats one long-context vector. Use long-context models for routing and triage, chunks for answers.

Then dimension and cost, last. Once you have a shortlist that satisfies residency, language, and modality, pick the smallest dimension that holds your recall on your evaluation set. Measure it: embed your evaluation corpus at full dimension and at a truncated dimension, compare recall@5, and take the smaller one if the gap is inside your noise band. This is a pure infrastructure savings with no user-visible downside when it holds.

Two closing rules. Always add a reranker before you upgrade the embedding model — a cross-encoder over the top 25 candidates typically buys more precision than any base-model swap, and it is a drop-in addition that does not require re-indexing. And always run hybrid keyword-plus-vector retrieval unless you have measured that pure vector wins on your queries; the combination is more robust to the exact-match failure mode that pure embeddings cannot fix.

Related questions

Do I need a vector database, or is a library enough?

Below roughly a hundred thousand vectors, an in-process library like FAISS or a Postgres extension handles search fine and removes an operational dependency. Managed vector databases earn their cost at larger scale, with metadata filtering, multi-tenancy, and horizontal scaling requirements.

How often should I re-embed my corpus?

Re-embed individual documents whenever their content changes — ideally triggered by your CMS or database change feed. Re-embed the entire corpus only when you change the model, the model version, the chunking strategy, or the preprocessing pipeline. Those four are the full-reindex triggers.

Can one embedding model serve search, clustering, and classification?

Usually yes. A single general-purpose model covers all three adequately, and running one model simplifies infrastructure considerably. Some models accept task-specific instruction prefixes that tune the same weights for retrieval versus clustering, giving task specialization without maintaining separate models.

What is a reranker and do I need one?

A reranker is a cross-encoder that reads the query and one candidate document together and scores relevance directly, rather than comparing two independently computed vectors. It is far more accurate and far slower, so you apply it only to the top 20–50 vector-search candidates. Most production RAG systems benefit.

Does a bigger embedding model always retrieve better?

No. Larger models generally score higher on benchmarks, but the gains narrow on narrow domains and can vanish entirely on jargon-heavy corpora where a fine-tuned smaller model wins. Larger models also cost more in storage, latency, and compute. Measure on your data before assuming.

FAQ

What exactly does an embedding model output?

A fixed-length list of floating-point numbers — commonly 384 to 4096 values — representing the input's position in a learned semantic space. The individual numbers are not human-interpretable; only distances between vectors carry meaning. Cosine similarity between two vectors is the standard way to measure how related their inputs are.

Can I mix vectors from two different embedding models in one index?

No. Each model defines its own coordinate space, so a vector from one model is meaningless relative to a vector from another. Mixing them produces nonsense similarity scores. If you want the benefit of multiple models, run separate indexes and fuse the ranked result lists, which is a different and valid technique.

How do I know if my retrieval is actually working?

Build a labeled evaluation set: 100–300 real user queries paired with the document IDs that correctly answer them. Measure recall@5 and recall@20 — the share of queries where a correct document appears in the top 5 or 20 results. Track this on every pipeline change. If recall@20 is high but recall@5 is low, add a reranker rather than swapping models.

Should I fine-tune an embedding model on my domain?

Only after exhausting cheaper options — better chunking, hybrid search, a reranker, and metadata filtering. Fine-tuning requires labeled query-document pairs, ongoing maintenance, and a full re-index on every retrain. It pays off for genuinely idiosyncratic domains with heavy internal jargon, and rarely pays off otherwise.

Why does my semantic search miss exact product codes?

Embeddings capture meaning, not string identity, so visually similar identifiers land near each other and distinct ones may not separate. Run BM25 keyword search in parallel and fuse the results with reciprocal rank fusion. Hybrid retrieval is the standard fix and is worth building in from day one rather than retrofitting.

Where does the embedding model fit relative to the LLM in a RAG stack?

The embedding model runs before the LLM and determines what the LLM sees. It converts documents into an index offline and converts each query into a lookup at request time. The LLM never touches your corpus directly — it only reads whatever passages retrieval handed it, which is why embedding quality bounds answer quality.

Sources

• OpenAI — Embeddings guide
• MTEB leaderboard on Hugging Face
• Cohere — Embed documentation
• Google Cloud Vertex AI — Text embeddings
• Pinecone — Vector database learning center
• Weaviate — Documentation
• Sentence-Transformers documentation
• Facebook Research — FAISS
• BEIR retrieval benchmark
• pgvector — Postgres vector extension

Related on PULSE

• [What is the role of Kubernetes in modern AI infrastructure?](/knowledge/ai431)
• [The 10 Best Embedding Models for Search and RAG in 2027](/knowledge/ai362)
• [How do you secure an LLM application's infrastructure?](/knowledge/ai363)
• [What infrastructure do you need to run AI agents in production?](/knowledge/ai373)
• [What infrastructure do you need for fine-tuning versus RAG?](/knowledge/ai427)
• [What is the difference between batch and real-time inference infrastructure?](/knowledge/ai409)