How do you evaluate LLM output quality at scale?

The outcome you should expect

A working evaluation program does not produce a single number that tells you your model is "good." It produces a scoring pipeline whose agreement with trusted human judgment is measured, known, and stable enough that you can act on score movements without re-reading the outputs yourself. That is the actual deliverable, and it changes what the team can do.

Concretely, expect three things once the pipeline is running. First, every output gets a score within minutes of generation instead of days, so a prompt regression surfaces on the same day it ships rather than in a customer complaint three weeks later. Second, human reviewer time collapses onto the ambiguous middle — the outputs where the automated judge is genuinely uncertain — which typically means reviewing 5–15% of volume instead of 100%, at a fraction of the cost per decision. Third, model and prompt changes become comparable: you can run version A and version B against the same frozen evaluation set and get a difference you can defend in a review meeting.

The correlation number is the honest headline. Published work on LLM-as-judge methods, notably the G-Eval paper (Liu et al., 2023), reports Spearman correlations with human raters in the 0.5–0.6 range on summarization benchmarks — substantially better than n-gram overlap metrics like BLEU and ROUGE, which typically land in the 0.2–0.4 band on the same tasks, but nowhere near the ceiling. Your own number will depend heavily on task, rubric clarity, and how well your human labels agree with each other. If your human raters only agree with each other 70% of the time, no judge will exceed that ceiling, and chasing a higher automated correlation is chasing noise.

Set expectations accordingly with stakeholders. The pipeline gives you a reliable *relative* signal — this prompt is better than that one, this week is worse than last week — long before it gives you a trustworthy *absolute* one. Teams that treat an automated score of 4.2/5 as a literal quality certificate get burned. Teams that treat a drop from 4.2 to 3.8 as a trigger to go look at samples get value on day one.

There is a direct revenue argument here that survives scrutiny. If LLM outputs touch a customer-facing surface — support replies, sales email drafts, product descriptions, onboarding content — then quality variance is conversion variance. An evaluation pipeline that catches a bad prompt deploy in hours instead of weeks limits the blast radius to hundreds of interactions rather than tens of thousands. That is the number to put in the business case, not the correlation coefficient.

What drives that outcome

Four inputs determine whether your evaluation program produces a trustworthy signal or an expensive dashboard nobody believes.

Rubric specificity. A judge prompt that asks "rate this output 1–5 for quality" produces noise, because the judge is inventing its own definition of quality on every call. A rubric that names the dimension, defines each score level in concrete observable terms, and gives one anchor example per level produces far more stable scores. Split the rubric into separate criteria — factual accuracy, instruction-following, tone, completeness, safety — and score each with its own focused call rather than asking for five numbers in one response. The per-criterion approach costs more in API calls but reduces the cognitive load on the judge and makes disagreements diagnosable: you can see *which* dimension moved.

Gold-set quality. Your calibration set is the foundation everything else rests on. Build 200–500 human-labeled examples per task category, deliberately stratified — not a random sample of production traffic, which will be 90% easy cases. Include known-good outputs, known-bad outputs, and the ambiguous middle in roughly equal measure. Have at least two humans label each item and measure inter-rater agreement (Cohen's kappa or Krippendorff's alpha). If agreement is poor, the rubric is ambiguous, and you fix the rubric before you touch the judge.

Judge bias control. LLM judges carry documented systematic biases. Position bias: when comparing two outputs, judges favor whichever appears first — mitigate by running each pairwise comparison twice with the order swapped and discarding disagreements. Length bias: judges tend to reward longer, more elaborate responses independent of substance — check this directly by plotting score against output length on your gold set and looking for a slope that shouldn't be there. Self-preference: a judge model tends to score outputs from its own family more favorably, which matters if you're comparing vendors. Use a judge from a different family than the model under test where the comparison is competitive.

Sampling design. You cannot afford human review of everything and you don't need it. Stratify by output type, length, and risk tier, then sample within strata. For a 95% confidence level with a 5% margin of error on a proportion, you need roughly 385 samples per stratum — that is the standard sample-size arithmetic, and it holds regardless of how large the underlying population is.

The feedback edge from human-labeled outcomes back into the gold set is the part teams skip, and it is the part that keeps the pipeline honest as traffic drifts. Without it, your judge is calibrated against a snapshot of last quarter's inputs.

Benchmarks and realistic ranges

Cost is the first thing leadership asks about, so anchor it in the actual arithmetic rather than a remembered per-output figure. An LLM-as-judge call costs whatever your provider charges for the tokens it consumes: the rubric prompt, the output being judged, and a short structured verdict. For a five-criterion rubric on a 500-token output, budget roughly 4,000–8,000 input tokens and a few hundred output tokens per evaluation across all criteria. Multiply by your provider's current published per-million-token rate — check the vendor's pricing page rather than trusting a number in a blog post, because these rates have fallen repeatedly and any figure written down goes stale within months.

Two levers cut that materially. Batch APIs from major providers offer a substantial discount (commonly around 50%) for asynchronous jobs with a turnaround window measured in hours — evaluation is an ideal batch workload since nothing downstream blocks on it. Prompt caching helps even more: the rubric and few-shot anchors are identical across every call, so caching that prefix cuts the repeated input cost sharply. Between the two, a naive per-output cost often drops by 60–80% with no change to the scoring logic.

Human review costs are the dominant line item and scale linearly. A trained reviewer working a clear five-dimension rubric handles an output in roughly 2–5 minutes depending on length and domain complexity; specialist reviewers in medical, legal, or financial domains run longer and cost considerably more per hour. Take your loaded hourly rate, divide by outputs per hour, and you have a per-review figure you can defend. The point of the tiered pipeline is to shrink the count that hits this line, not the rate.

Latency ranges are worth planning around. Judge-based scoring adds seconds per output and is unsuitable for synchronous, in-request gating. Embedding-similarity metrics like BERTScore run in milliseconds on GPU and cost effectively nothing at inference, which makes them viable as a real-time first-pass filter — but they need reference outputs, and their agreement with human judgment degrades sharply on open-ended creative tasks where many different outputs are equally valid. Use cheap metrics to triage and expensive judges to decide.

For correlation expectations by method family: n-gram overlap metrics (BLEU, ROUGE) are weakest and should not anchor a quality program; embedding-based metrics (BERTScore, BLEURT) improve meaningfully on tasks with well-defined references; LLM-as-judge with a structured rubric outperforms both on open-ended generation. Ensembles of multiple judge models generally beat any single judge, at multiplied cost. Do not treat any published correlation figure as a promise about your workload — reproduce it on your own gold set before you build on it.

Consistency-based hallucination detection deserves its own budget line. SelfCheckGPT-style methods sample the same prompt several times at nonzero temperature and check whether the sampled responses entail each other; the original work (Manakul et al., 2023) demonstrates this as a zero-resource, black-box approach requiring no external knowledge base. The cost is straightforward: 5–10 additional generations per output being checked, so five to ten times your base generation cost for anything you route through it. Reserve it for factual claims in high-stakes surfaces, not for everything.

Risks, edge cases, and failure modes

Gold-set staleness. The most common silent failure. You calibrate against 300 examples in January, traffic shifts in March, and by June the judge is well-calibrated for a distribution that no longer exists. Refresh 10–20% of the gold set quarterly with recent production samples, and re-measure judge-human correlation each time. If correlation drops more than a few points, stop trusting the dashboard until you've diagnosed why.

Overfitting to the judge. Once a team knows the score is what gets measured, prompts get tuned to please the judge. If the judge rewards length, prompts grow longer. If it rewards hedged, comprehensive-sounding answers, outputs get hedged. The score climbs; user satisfaction does not. Guard against it with a holdout evaluation set the prompt engineers never see, and with a periodic human-only review that samples outputs blind to their automated scores.

Judge model version drift. Provider-hosted models change under a stable-looking name. If your evaluation pipeline calls a floating alias, a silent upgrade can shift every score in your history and make quarter-over-quarter comparisons meaningless. Pin explicit model versions in the evaluation path. When you must upgrade, run both versions over a few hundred gold examples first and quantify the shift so you can annotate the dashboard rather than mistaking a judge change for a quality change.

Temperature and nondeterminism. Set the judge to temperature 0 to make repeat runs as stable as possible, but do not assume perfect determinism — hosted inference can vary run to run for reasons outside your control. Score a fixed set of 50 outputs twice a week and track the variance; that number is your noise floor, and any observed change smaller than it is not a signal.

Safety and correctness are not "quality." A fluent, well-structured, on-tone response that states something false is scored highly by fluency-oriented metrics. Perplexity in particular measures fluency, not truth, and will happily reward confident nonsense. Any evaluation program touching factual claims needs a separate factual-consistency check with its own threshold and its own escalation path — never a single blended score that lets fluency mask a hallucination.

Rater drift and fatigue. Human raters are not a stable measuring instrument either. Agreement degrades over long sessions, and individual raters drift toward leniency or severity over weeks. Insert known-answer control items at roughly 1-in-20 frequency, monitor per-rater accuracy on those controls, and rotate reviewers off a task before quality decays. Track inter-rater agreement as an ongoing metric, not a one-time setup number.

Rare-but-severe failures. Random sampling at 5% will reliably miss a failure mode that occurs in 0.1% of outputs but causes real harm when it does. Sampling is the wrong instrument for that class of problem. Pair it with deterministic rule-based checks — regex and classifier gates for PII leakage, prohibited claims, competitor mentions, or policy violations — that run on 100% of outputs at negligible cost. Statistical sampling tells you about the average; rules catch the tail.

Cost runaway. Multi-judge ensembles plus consistency sampling plus per-criterion scoring can quietly cost more than generating the outputs did. Set an evaluation budget as an explicit percentage of inference spend, instrument it, and alert when it drifts. If evaluation exceeds roughly 10–20% of generation cost, you are probably over-evaluating easy cases that a cheap filter should have cleared.

A practical rollout plan

Roll this out in four phases over roughly eight to twelve weeks. Do not attempt to build the full tiered pipeline before you have proven the judge agrees with your humans on anything.

Phase one — define and label (weeks 1–3). Write the rubric before you write any code. Pick three to five dimensions that matter for your specific task and define each score level in observable terms. Then build the gold set: 200–500 stratified examples, two independent human labels each, inter-rater agreement measured. Expect the first pass to expose rubric ambiguity — a dimension where your two raters systematically disagree is a dimension you have not defined well enough. Rewrite and re-label. This phase feels slow and is the highest-leverage work in the project.

Phase two — calibrate the judge (weeks 3–5). Implement judge scoring against the gold set only, not production. Measure Spearman correlation and per-dimension agreement. Run the bias checks: score-versus-length regression, order-swapped pairwise comparisons, and a self-preference check if the judge and the model under test share a family. Iterate on the judge prompt until correlation plateaus. Record the final correlation figure as the pipeline's stated accuracy — publish it alongside every dashboard so nobody over-reads the scores.

Phase three — shadow and tier (weeks 5–8). Run the calibrated judge over production traffic in shadow mode: score everything, act on nothing. Use two weeks of shadow scores to set your confidence thresholds empirically. Find the upper band where the judge's high-confidence passes agree with human review often enough to auto-accept, and the lower band where escalation is warranted. Then turn on tiered routing: auto-accept above the upper threshold, single reviewer in the middle, multi-reviewer panel below the lower one. Expect the middle band to start wide and narrow as the rubric sharpens.

Phase four — operate and defend (week 8 onward). Wire scores into the same alerting your service metrics use, with alerts on distribution shift rather than single-output scores. Add the holdout set, the rater control items, and the quarterly gold-set refresh to a recurring calendar. Pin judge model versions and treat a judge upgrade as a change requiring its own before/after measurement.

Two rollout details make the difference between a pipeline that survives and one that gets abandoned. First, store every score with the full context that produced it — prompt version, model version, judge version, rubric version, timestamp. Without that, you cannot answer "did quality drop or did we change how we measure?" six months from now, and that question will be asked. Second, give the pipeline an owner. Evaluation infrastructure with no owner rots faster than almost any other kind, because it degrades invisibly: the dashboard keeps rendering numbers long after those numbers stopped meaning anything.

Related questions

How large should the human-labeled gold set be?

Start with 200–500 examples per task category, stratified across output types and difficulty. Below roughly 200, correlation estimates are too noisy to act on. Above 500, returns diminish quickly — spend the extra budget on refreshing the set quarterly rather than on making it bigger once.

Should the judge model differ from the model being evaluated?

Yes when the comparison is competitive. LLM judges show measurable self-preference toward outputs from their own model family, which biases vendor comparisons. For monitoring a single production model over time, same-family judging is acceptable since the bias is constant and you're reading relative movement.

Can automated evaluation fully replace human review?

No. It can shrink human review from 100% of volume to the ambiguous band and the high-risk tier, typically 5–15%. Keeping a human sample is also what lets you detect judge drift — remove it entirely and you lose the only independent check on whether your automated scores still mean anything.

What is the single most common mistake teams make?

Skipping calibration. Teams stand up an LLM judge, get plausible-looking scores, and start making decisions without ever measuring agreement with human judgment. The scores look authoritative and may be near-random for their specific task. Measure correlation first, publish it, and never let a dashboard imply more precision than that number supports.

How do you catch rare failures that sampling misses?

Deterministic rule-based checks on 100% of outputs — regex and classifier gates for PII, prohibited claims, or policy violations. These cost almost nothing and catch the tail that a 5% sample will systematically miss. Statistical sampling measures the average; rules catch the outliers.

FAQ

How do I detect hallucinations in LLM output at scale?

Use a consistency-based check: sample the same prompt several times at nonzero temperature and test whether the responses entail one another. Inconsistency across samples is a strong hallucination signal and requires no external knowledge base — the SelfCheckGPT work established this as a black-box, zero-resource approach. The cost is 5–10 extra generations per checked output, so reserve it for factual claims on high-stakes surfaces rather than applying it to all traffic.

Can one metric cover every evaluation need?

No. Different metrics capture different dimensions, and blending them into one number lets a strength mask a weakness — fluent text scores well on perplexity while being factually wrong. Run a composite: an LLM judge for coherence and instruction-following, a consistency check for factual claims, and reference-similarity metrics where good reference outputs exist. Weight the components by what actually matters for your task and report the components alongside the composite.

How do I know whether my judge is biased?

Test for it explicitly on the gold set. Plot score against output length and look for a slope that substance alone doesn't explain. Run pairwise comparisons twice with the order reversed and count how often the verdict flips — flips indicate position bias. If the judge and the evaluated model share a family, compare its scores against a judge from a different family. Each of these is a few hours of work and each catches a bias that would otherwise silently distort months of data.

What should I do when judge scores and human scores disagree?

Treat the disagreement as information about the rubric, not as an error to suppress. Pull twenty disagreement cases and read them. Usually one of three things is true: the rubric is ambiguous on that dimension, the human raters themselves disagree there, or the judge is hitting a known bias. Fix the underlying cause. Never tune the judge to match humans on the same examples you use to measure agreement — that overfits and inflates your reported correlation.

How often should I recalibrate?

Quarterly at minimum, and immediately after any change to the judge model version, the rubric, or the upstream prompt. Refresh 10–20% of the gold set with recent production samples each cycle and re-measure correlation. A drop of more than a few points means the input distribution has shifted and your thresholds need resetting before the dashboard is trustworthy again.

Does this scale down to a small team?

Yes, and the tiered approach is where small teams get the most leverage. A single engineer can build a working judge pipeline in a few days; the expensive part is the gold set, and 200 carefully chosen examples labeled by two people is achievable in under a week. Skip the multi-judge ensemble and the consistency sampling initially — one calibrated judge plus rule-based safety gates plus a small weekly human sample covers most of the value.

Sources

• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative LLMs
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• BERTScore: Evaluating Text Generation with BERT
• Hugging Face Evaluate Library Documentation
• Label Studio — Open Source Data Labeling Platform
• Stanford HELM: Holistic Evaluation of Language Models
• NIST AI Risk Management Framework
• OpenAI Evals

Related on PULSE

• [How do you scale LLM inference to handle thousands of concurrent users?](/knowledge/ai347)
• [How do you secure an LLM application's infrastructure?](/knowledge/ai363)
• [How do you build a cost dashboard for AI and LLM spend?](/knowledge/ai417)
• [What is the best way to cache embeddings at scale?](/knowledge/ai419)
• [The 10 Best LLM Gateways in 2027](/knowledge/ai354)
• [The 10 Best LLM Routing and Load Balancing Tools in 2027](/knowledge/ai412)