How do you set up observability for a RAG application?

The outcome you should expect

The point of RAG observability is not a dashboard. It is the ability to answer, within minutes of a user complaint, the single question that matters: *did the retriever fail, or did the generator fail?* Before instrumentation, that question takes hours of manual reproduction and usually ends in a shrug. After instrumentation, you open a trace, look at the retrieved chunks, read the assembled prompt, and see immediately whether the correct passage was in context and the model ignored it, or whether the correct passage never made it into context at all. Those two failure modes have completely different fixes — one is a prompt, reranker, or model problem; the other is a chunking, embedding, or index problem — and without traces teams routinely spend a week tuning the wrong half of the system.

A realistic post-instrumentation state looks like this. Every request emits a trace with six to twelve spans. Each retrieval span records the query string (and the rewritten query, if you rewrite), the top-k chunk IDs, their similarity scores, the source documents, and the vector database round-trip time. Each generation span records the model, the full assembled prompt, the completion, prompt and completion token counts, and the computed dollar cost. A parent span records end-to-end latency and the final user-visible response. On top of that, a sampled subset — commonly 5% to 20% in production, 100% in staging — gets asynchronous evaluation: an LLM-as-judge or heuristic scorer computes faithfulness (is the answer grounded in the retrieved context?), answer relevance (does it address the question?), and context precision (what fraction of retrieved chunks were actually useful?).

The second outcome is a defensible cost model. RAG applications have a nasty habit of quietly tripling their bill. Someone bumps top-k from 3 to 8 to "improve recall," and every request now carries five extra chunks of context. If chunks average 400 tokens, that is 2,000 extra input tokens per query. At a million queries a month, the difference between a 1,500-token and a 3,500-token average prompt is enormous, and nothing in your application logs will tell you it happened. Token counting per span makes that change visible on the day it ships, attributed to the exact deploy and the exact configuration change.

The third outcome is regression safety. Once traces exist, they become datasets. You curate 50 to 300 real production queries — especially the ones that failed — into an evaluation set, and every pull request that touches chunking, prompts, retrieval parameters, or the model runs against it. That converts RAG development from vibes-driven tinkering into something with a pass/fail gate, which is the actual difference between a demo and a system.

Expect the full setup to take a competent engineer somewhere between two days and two weeks depending on how custom the pipeline is. A stock LangChain or LlamaIndex pipeline with an off-the-shelf integration is close to a one-line instrumentation change. A hand-rolled pipeline with a custom reranker, a hybrid BM25-plus-vector retriever, and a multi-hop agent loop takes longer, because you are writing the spans yourself.

What drives that outcome

Four things determine whether your observability setup is genuinely useful or just expensive log storage.

Span granularity. The most common mistake is a single span per request. It tells you a request was slow and gives you no idea why. The right granularity is one span per meaningful operation, and in a RAG pipeline those operations are: query preprocessing or rewriting, query embedding, vector search, optional keyword or hybrid search, reranking, deduplication and context assembly, prompt templating, LLM generation, and any post-processing or citation extraction. Nine spans is not excessive. Each one should carry attributes, not just timing — the retrieval span without the chunk IDs and scores is nearly worthless.

Semantic conventions. OpenTelemetry has emerging semantic conventions for generative AI (attributes in the gen_ai.* namespace: gen_ai.system, gen_ai.request.model, gen_ai.usage.input_tokens, and so on). Adopting them costs nothing at instrumentation time and buys you portability. Teams that invent their own attribute names find that every observability tool they try renders their traces as opaque blobs, because the tools key their specialized RAG views off known attribute names. If you want the retrieval-quality UI, the embedding projector, and the cost dashboards to work out of the box, emit the attributes those views expect.

Evaluation coupled to traces. Metrics that live separately from traces decay into wall art. The value comes from a faithfulness score attached to the specific trace whose retrieved chunks and generated answer you can immediately open and read. That is what makes triage fast: sort by lowest faithfulness, open the worst five traces, and the pattern is usually obvious within ten minutes — a document type that chunks badly, a query phrasing the embedding model handles poorly, a prompt instruction the model keeps overriding.

Sampling strategy. You cannot afford to evaluate every request with an LLM judge; a judge call often costs as much as the original generation. But you also cannot sample uniformly and expect to catch rare failures. The practical pattern is stratified: 100% of requests get traced (traces are cheap), a uniform baseline slice (say 5%) gets evaluated for trend tracking, and then you force evaluation on interesting cases — requests where the top similarity score was below a threshold, where the user gave negative feedback, where the response contained hedging language, where latency exceeded p95, or where zero chunks passed the relevance cutoff. That biased sample is where the bugs live.

Notice what the second diagram implies about instrumentation requirements. To walk that decision tree you need the chunk IDs, the similarity scores, the pre-rerank and post-rerank ordering, and the final assembled context — all four. Drop any one and a branch of the tree becomes unanswerable. That is the concrete argument for rich span attributes over minimal ones.

The tooling landscape splits along a predictable axis. Purpose-built LLM observability platforms — Arize Phoenix, LangSmith, Langfuse, Weights & Biases Weave, Langtrace — ship with retrieval-aware views, prompt playgrounds, and evaluation runners. General APM platforms — Datadog, SigNoz, Grafana with Tempo, Honeycomb — give you mature alerting, long retention, and correlation with infrastructure metrics, but expect you to define what a "retrieval span" means. Several of the purpose-built tools are OpenTelemetry-native, which means the choice is less locked-in than it looks: instrument once against OTel, export to whichever backend, and switch later by changing an exporter endpoint. That is a materially better position than adopting a proprietary SDK that hard-codes your instrumentation to one vendor's schema.

Cost structures differ meaningfully. Open-source options like Phoenix, Langfuse, Langtrace, and SigNoz can be self-hosted at infrastructure cost only, which for a moderate-volume application is often a single container and a database. Managed tiers typically price on trace or span volume, sometimes on seats. General APM tools usually price on hosts plus ingested data. The relevant question is not the sticker price but the volume math: a chatty RAG agent that makes six LLM calls per user turn generates six times the spans of a single-shot pipeline, and span-priced plans notice.

Benchmarks and realistic ranges

Treat every number here as a starting reference, not a target — the right values depend on corpus size, document type, and how tolerant your users are of latency.

Latency budgets. For an interactive chat-style RAG application, a reasonable end-to-end p50 sits in the 1.5 to 3 second range for a non-streaming response, with time-to-first-token under about 1 second when streaming. Within that budget, query embedding is typically 20 to 100 ms for a hosted embedding API and single-digit milliseconds for a local model. Vector search against a well-tuned index of a few million vectors commonly lands in the 10 to 100 ms range; if you are seeing 500 ms, either the index is unwarmed, the filter is forcing a scan, or you are hitting a cold serverless tier. Cross-encoder reranking of 20 to 50 candidates is often 50 to 300 ms depending on whether it runs locally or as an API call. Generation dominates the rest, and it scales with output length far more than input length. The practical implication: if your p95 blows out, check output token counts before you blame the vector database.

Retrieval parameters. Chunk sizes in production RAG systems commonly land between 256 and 1,024 tokens with 10% to 20% overlap, though structured documents often do better with structure-aware splitting (by section, by heading, by table) than by fixed token count. Retrieval top-k is frequently 3 to 10 for direct answering; if you rerank, a common pattern is retrieve 20 to 50 candidates and pass the top 3 to 5 forward. Every increment of top-k costs you input tokens linearly, so the cost/quality trade-off is visible and measurable — this is exactly the experiment your evaluation set exists to settle.

Quality metrics. Faithfulness and answer-relevance scores are usually normalized to a 0–1 range. Mature systems on well-curated corpora often report faithfulness in the 0.85 to 0.95 band; anything persistently below about 0.7 signals real grounding problems. Context precision — the fraction of retrieved chunks that were actually relevant — is frequently much lower than teams expect, often 0.3 to 0.6 with naive top-k retrieval, which is precisely why reranking earns its latency. Absolute values matter far less than the delta across deploys. A faithfulness drop from 0.91 to 0.84 after a chunking change is a strong, actionable signal; the raw 0.84 in isolation is not.

Evaluation set size. Fifty queries is enough to catch gross regressions. Two hundred to three hundred gives you enough resolution to detect a few-percent shift with some confidence. Beyond that, the marginal value falls off quickly relative to the cost of running judge calls on every pull request. A pragmatic split: a fast tier of 30 to 50 queries that runs on every PR in a minute or two, and a full tier of 200 to 500 that runs nightly or before release.

Cost per query. A single RAG turn typically involves one embedding call (cheap — embeddings are one to two orders of magnitude cheaper per token than generation) and one generation call. The generation dominates. The dangerous multiplier is agentic RAG: a loop that retrieves, reasons, retrieves again, and re-reads its own context can easily cost five to ten times a single-shot query, and it does so invisibly unless your traces aggregate cost at the parent-trace level rather than per-call. Set a cost-per-trace alert, not just a cost-per-call alert.

Storage and retention. Full-fidelity traces including prompts and completions are large — an individual RAG trace with ten retrieved chunks can be tens of kilobytes. At meaningful volume, retention policy becomes a real decision. A common arrangement is 7 to 30 days of full-fidelity traces for debugging, with metrics and evaluation scores rolled up and retained for a year or more for trend analysis. Curated evaluation datasets get retained indefinitely because they are the crown jewels.

Risks, edge cases, and failure modes

You are logging sensitive data. This is the risk that hurts. Full-prompt tracing means you are storing whatever the user typed and whatever documents your retriever pulled, which in a healthcare, legal, financial, or HR context is exactly the data you have compliance obligations around. Decide this before you instrument, not after. Options in rough order of preference: redact PII at the SDK level before the span leaves your process; hash or reference document content rather than storing it inline; self-host the observability backend inside your own trust boundary; or restrict full-fidelity capture to a staging environment with synthetic data. Storing raw production prompts in a third-party SaaS without a data processing agreement and a documented retention window is a genuine audit finding waiting to happen.

Instrumentation overhead in the hot path. Span creation is cheap; synchronous export is not. If your exporter blocks on a network call per span, you have added tens of milliseconds to every request and coupled your application's availability to your observability vendor's. Use batched, asynchronous export with a bounded queue and a drop policy. Verify the failure behavior explicitly: kill the collector in staging and confirm your application still serves requests. More than one team has discovered during an incident that their observability layer was the thing amplifying the outage.

LLM-as-judge is itself unreliable. Judge scores are noisy, sensitive to prompt phrasing, and biased in known ways — toward longer answers, toward answers that echo the question's wording, toward outputs from the same model family as the judge. They are useful as a relative signal across deploys and nearly useless as an absolute quality claim. Calibrate the judge against a few dozen human-labeled examples before you trust it, re-check that calibration when you change judge models, and never let a judge score alone block a release without a human glance at the failing cases.

Silent index drift. Your corpus changes: documents get added, updated, deleted, or re-embedded. If you change embedding models without fully re-indexing, you get a corpus where old and new vectors live in incompatible spaces and retrieval quality collapses in a way that is very hard to see from application logs. Monitor index freshness (age of newest document), index size (a sudden drop means a failed sync), embedding model version as an explicit span attribute, and the distribution of top-1 similarity scores over time. A shifting score distribution is often the earliest warning that something upstream broke.

Query distribution shift. Users change what they ask. A retrieval setup tuned on the questions you imagined can degrade badly when real usage skews toward a different intent — say, from factual lookup to multi-document comparison. Cluster query embeddings and watch the cluster distribution weekly. New clusters that correlate with low similarity scores and low faithfulness point to a coverage gap in your corpus, not a bug in your retriever.

The zero-results and near-miss cases. Every RAG system has queries where nothing relevant exists. The failure mode is a confident, fluent, entirely fabricated answer. Instrument explicitly for this: record the top similarity score, record how many chunks cleared your relevance threshold, and alert when the rate of low-confidence retrievals rises. Then make sure the application actually behaves differently in that case — declining to answer is a feature, and you need observability to know how often it should be firing versus how often it is.

Streaming complicates everything. With streamed responses, the span does not close when the first token arrives. You need to record both time-to-first-token and total generation time, and handle the case where the client disconnects mid-stream and the span never properly closes. Unclosed spans produce ugly gaps in your latency percentiles. Set explicit timeouts and finalize spans in a finally block.

Multi-tenant leakage. If your RAG application serves multiple customers, every span needs a tenant identifier, and your observability access controls need to respect it. This matters twice: for compliance, and for debugging, because "retrieval quality is bad" is often really "retrieval quality is bad for the one tenant whose documents are all scanned PDFs."

Cardinality explosions. Putting the raw query string into a metric label rather than a span attribute will generate unbounded cardinality and either bankrupt you or take down your metrics backend. Queries, user IDs, and document IDs belong on spans. Metrics labels should be bounded dimensions: model, tenant tier, retrieval strategy, environment.

The adjacent case worth planning for. Everything above generalizes to agentic and tool-calling systems, which are increasingly what RAG applications become. Once your pipeline includes a loop that decides whether to retrieve again, the trace becomes a tree rather than a chain, and you need to record the decision at each step — why the agent chose to search again, what it searched for, when it stopped. The instrumentation principles are identical; the volume and the depth are not. Building your span hierarchy properly now means the agentic version is an extension rather than a rewrite.

A practical rollout plan

Do this in stages. Trying to build the full stack in one sprint usually produces something nobody trusts.

Stage one — trace the happy path, week one. Pick your backend and get one end-to-end trace rendering with correct parent-child nesting. Instrument only three spans initially: retrieval, generation, and the parent request. Use an existing integration if your framework has one; write manual spans if not. The deliverable is unglamorous and essential — a single trace you can click into and read. Verify it in a deployed environment, not just locally, because collector networking is where the surprises live.

Stage two — enrich attributes, week one to two. Now add the details that make traces diagnostic: chunk IDs, similarity scores, source document references, the rewritten query, the assembled prompt, model name, token counts, computed cost, and a tenant or user identifier. Add environment and release-version attributes to everything — without them, you cannot attribute a regression to a deploy, which is half the reason you are doing this. Split retrieval into its real sub-steps at this stage: embedding, search, rerank, assembly.

Stage three — build the evaluation set, week two to three. Pull 50 to 200 real queries from your now-existing traces. Deliberately over-sample failures and edge cases. Write expected answers or at least acceptance criteria for each. This is manual work and there is no way around it; budget a day or two of a domain expert's time. Store the set in version control alongside your code so it changes through review like everything else.

Stage four — wire evaluations, week three. Run faithfulness, answer relevance, and context precision scorers against the set. Calibrate the judge against human labels on a subset. Establish your current baseline and write it down. Then turn on sampled production evaluation with the stratified strategy described earlier.

Stage five — gate CI, week three to four. Add a pull-request check that runs the fast evaluation tier when retrieval, prompt, chunking, or model code changes. Set thresholds relative to the current baseline rather than absolute — for example, fail if faithfulness drops more than 5 points or p95 latency rises more than 20%. Make the failure output show the specific queries that regressed, because a bare "eval failed" gets ignored within two weeks.

Stage six — alerting and review cadence, ongoing. Alert on things that indicate breakage rather than noise: error rate, p95 latency, retrieval returning zero results above a threshold rate, cost per hour exceeding a ceiling, and index staleness. Send quality-score drift to a weekly review rather than a pager — quality degrades gradually and paging on it just trains people to ignore the pager. Hold a short weekly session where someone actually opens the ten worst-scoring traces and reads them. That habit finds more real problems than any dashboard.

Stage seven — close the user feedback loop. Add thumbs up/down to the interface and attach the rating to the trace ID. Human feedback is the only ground truth you have, and correlating it against your automated scores tells you whether your judge is measuring anything real. Negative-feedback traces should flow automatically into your evaluation set candidate pool.

A note on sequencing that teams get wrong: do not build dashboards before stage three. Dashboards without an evaluation baseline show you numbers with no reference point, and people stop looking at them. The trace-reading habit comes first; the aggregate views become useful only once you know what a good number looks like.

One more adjacent consideration. If your organization already runs a mature observability practice for conventional services, resist the urge to build the RAG stack entirely in isolation. Exporting LLM traces into the same backend your infrastructure traces live in lets you correlate a retrieval latency spike with the vector database pod that was evicted, or a generation error rate with an upstream gateway change. The purpose-built tools give you better RAG-specific views; the shared backend gives you better causal reasoning across the whole system. Many teams end up running both, with OpenTelemetry fanning out to two exporters — which is entirely reasonable and costs you one configuration block.

Related questions

How is RAG observability different from standard APM?

Standard APM tracks latency, errors, and throughput — all deterministic. RAG adds non-deterministic quality: the same query can return a good answer today and a bad one tomorrow with identical latency. You need semantic evaluation of retrieved context and generated output alongside conventional timing spans.

Do I need OpenTelemetry, or will a vendor SDK do?

A vendor SDK is faster to start. OpenTelemetry is safer long-term because it decouples instrumentation from backend, letting you switch tools by changing an exporter. Several LLM observability platforms are OTel-native, so you often get both. Prefer OTel for anything you expect to run for years.

How do I measure retrieval quality without labeled data?

Start with proxy signals available immediately: top-1 similarity scores, score distribution over time, and rate of zero-result queries. Then use LLM-as-judge for context precision and faithfulness, which needs no ground truth. Curate labels gradually from real failures rather than trying to label upfront.

What should I alert on versus review weekly?

Page on breakage — errors, p95 latency, cost ceilings, index staleness, zero-result rate spikes. Review weekly on drift — faithfulness scores, context precision, query distribution shifts, feedback ratios. Quality degrades gradually; paging on it produces alert fatigue and trains people to dismiss real signals.

Does observability slow down my RAG application?

Properly configured, negligibly — span creation is microseconds and export is batched asynchronously. Problems come from synchronous export or inline LLM-judge evaluation in the request path. Keep evaluation asynchronous and off the hot path, and test that your app survives the collector being unreachable.

FAQ

Where do I start if my RAG pipeline is completely uninstrumented?

Add three spans: parent request, retrieval, and generation. Get one trace rendering correctly in a deployed environment with proper parent-child nesting. That single working trace is worth more than a week of planning, because it immediately surfaces whether your context propagation works across async boundaries — the thing that quietly breaks most first attempts.

How much of my traffic should I evaluate with an LLM judge?

Trace 100% — traces are cheap. Evaluate a stratified sample: a uniform baseline of roughly 5% for trend tracking, plus forced evaluation on interesting cases such as low similarity scores, negative user feedback, high latency, or zero chunks passing your relevance threshold. Uniform-only sampling misses the rare failures that matter most.

Can I use my existing Datadog, Grafana, or SigNoz setup instead of a dedicated LLM tool?

Yes, particularly if you instrument with OpenTelemetry. You get mature alerting, long retention, and correlation with infrastructure metrics. What you give up is the purpose-built retrieval views, prompt playgrounds, and built-in evaluation runners, which you would then build yourself. Many teams export to both — one config block, two exporters.

What is the biggest mistake teams make with RAG observability?

Recording timing without content. A retrieval span that says "took 45 ms" and nothing else cannot tell you whether the right chunk was retrieved, which is the only question that matters during triage. Chunk IDs, similarity scores, and the assembled context are the payload; the duration is a footnote.

How do I handle prompts and retrieved documents containing sensitive data?

Decide before you instrument. Redact PII at the SDK level so it never leaves your process, store document references rather than inline content where feasible, self-host the backend inside your trust boundary, and set an explicit short retention window on full-fidelity traces. Get a compliance sign-off recorded before production capture is enabled.

How do I stop RAG costs from creeping up unnoticed?

Record prompt and completion tokens on every generation span and compute cost as an attribute. Aggregate at the parent-trace level, not per-call — agentic loops with multiple retrieval rounds hide their real cost otherwise. Then alert on cost per trace and cost per hour, and correlate spikes against release-version attributes to attribute them to a deploy.

Sources

• OpenTelemetry Semantic Conventions for Generative AI
• Arize Phoenix Documentation
• LangSmith Documentation
• Langfuse Documentation
• Ragas — Evaluation Framework for RAG Pipelines
• LlamaIndex Observability Guide
• OpenTelemetry Sampling Documentation
• SigNoz Documentation
• Datadog LLM Observability
• Weights & Biases Weave Documentation

Related on PULSE

• [The 10 Best RAG Frameworks in 2027](/knowledge/ai352)
• [The 10 Best Vector Databases for RAG in 2027](/knowledge/ai338)
• [The 10 Best AI Observability Platforms in 2027](/knowledge/ai364)
• [The 10 Best AI Observability Tools for RAG Pipelines in 2027](/knowledge/ai436)
• [How do you secure an LLM application's infrastructure?](/knowledge/ai363)