How I Evaluate Production RAG Systems
Most RAG evaluation I see measures one thing: given a question with a known answer, did the system produce something close to it? That metric is real, but it evaluates the happy path of a system whose value, in serious settings, lies in how it behaves off the happy path. A production RAG system is a chain of decisions: what to retrieve, whether the evidence suffices, what to claim, what to cite, when to refuse. An evaluation that scores only the final prose cannot tell you which decision failed, and an evaluation that cannot localise failure cannot make the system safer. Mine scores the layers.
Score the decision chain, not just the answer
The first layer is retrieval, evaluated on its own before anything downstream gets credit or blame. Did the evidence that exists for this question actually surface? Did irrelevant material ride along with it? A generation layer handed the wrong records can only be fluent about the wrong thing, and an evaluation that scores answers without checking retrieval will misattribute retrieval misses as reasoning failures, then “fix” the wrong component. For every golden question, the set records which evidence should have been retrievable, so a miss is measurable directly.
The second layer is answerability: given what was actually retrieved, was answering the right decision? This is where the system’s sufficiency judgment gets scored, in both directions. Answering on insufficient evidence is fabrication under pressure, the worst cell in the whole grid. Refusing on sufficient evidence is over-caution, a real cost that erodes the patience of every user who got a refusal they didn’t deserve.
The third layer is faithfulness and citation quality, and the two must not be conflated. Faithfulness asks whether each surfaced claim is supported by the records it cites, at claim granularity. Citation quality asks whether each citation actually carries its claim, because a response is not better for containing citations; it is better for containing citations that hold. An answer with confident prose and decorative citations should score worse than an honest partial answer, and an evaluation that rewards citation count achieves the opposite.
The golden set adjudicates outcomes, not prose
All of this needs a golden set: questions with adjudicated expected system outcomes, held out from anything the system was tuned on. Not expected paragraphs. Expected outcomes: answer with these claims supported, or refuse for this reason. The moment the gold standard is a reference paragraph, the evaluation collapses back into prose similarity, and every layer above disappears from view.
A meaningful share of the set must be unanswerable on purpose, and unanswerable in different ways, because each way exercises a different decision: evidence genuinely absent from the corpus; questions resting on a false premise the records contradict; ambiguous questions with more than one defensible reading; questions where evidence exists but cannot carry the requested conclusion, so the honest outcome is a partial answer or a refusal naming what’s missing; and questions whose time range or sources the corpus doesn’t fully cover, where the honest outcome is a scoped “insufficient coverage” rather than a guess. That last class matters more than it looks; the deeper argument about coverage and denominators is its own article.
The answerable half needs its own adversarial members: questions engineered so the relevant evidence exists but is hard to surface (phrased away from the document’s vocabulary, split across records). Those exist to expose retrieval failures specifically, and they only work if the set’s author fights the instinct to write questions whose evidence is easy to find. Whoever writes the golden set knows the corpus too well; adversarial authorship is a deliberate act.
Refusals are outcomes with a taxonomy, not a bucket
Because refusal is a first-class outcome, “refused” is never a single count. A refusal has a cause, and the causes are different defects with different owners: retrieval failed to surface evidence that exists (a search problem); the question was malformed or ambiguous (a task problem); the corpus genuinely lacks coverage (a data problem, and the correct refusal); the evidence is present but conflicting (an adjudication problem); or the system errored and dressed the failure as caution. An evaluation that lumps these together can tell you the refusal rate moved and nothing about why.
So refusal reasons come from a small fixed vocabulary, and the evaluation scores the reason as well as the act: the right refusal names the right gap. Two runs refusing the same question for different named reasons is a finding, not noise.
One meta-lesson from running this loop repeatedly: when the system refuses a question the set marked answerable, the defect is as often in the golden set as in the system. An ambiguous question whose “expected answer” encodes one arbitrary reading teaches the evaluation to punish honesty. Refusal disagreements are where the golden set itself gets debugged, and a set that never gets corrected by its own results isn’t being taken seriously.
Denominators are where evaluations quietly lie
Every aggregate score is a fraction, and the numerator gets all the attention while the denominator does the lying. Scores reported over “questions the system attempted” silently exclude the timeouts, the parse failures, the retrieval errors. Scores over “valid runs” exclude whatever was invalid, with “invalid” doing unexamined work.
The sharpest version of the trap is a metric that sounds rigorous: a faithfulness score computed only over the questions the system chose to answer. It can be superb while the system refuses half its workload, because the refusals never enter the fraction. The rule I hold is simple: every run reports against the full question set, and every question ends in an explicit state. Answered, refused with reason, errored, timed out. No question vanishes between the run and the report; if a run errors on a fifth of its input, the headline includes the fifth. The same discipline applies inside a single answer: claim-level scoring over all claims made, including the ones that cited nothing, because an answer scored “mostly faithful” can hide the one unsupported claim that happens to be load-bearing.
Judge blind, adjudicate disagreement, never average
Where scoring requires judgment (is this claim supported by that record?), the judge is blind: it sees the claim and the evidence, never which system version produced it, never the generator’s confidence, never the surrounding prose. A judge that knows which output is the candidate inherits every bias in the pipeline. This is the evaluation-side twin of the independence constraint on the faithfulness gate.
Two signals are deliberately not accepted as ground truth in this methodology. The generator’s self-reported confidence, because certifying output is not the generator’s job and I don’t treat its self-assessment as evidence about correctness. And self-consistency (sampling repeatedly, checking agreement), because agreement across samples reproduces stable errors as happily as stable truths; a model that has confidently learned the wrong thing agrees with itself. Both can be useful telemetry. Neither is a verdict.
When judges disagree (human with model, model with model), the disagreement is adjudicated explicitly: both positions recorded, a decision made and attributed, the basis stated. Averaging judge scores buries exactly the cases that matter, because disagreement clusters on the genuinely ambiguous items, and those define where the system’s real boundary sits.
Releases are compared by failure class, not by average
All of this exists to answer one operational question: does the new version (new model, new prompt, new index, new chunking) ship? A single aggregate score cannot answer it, because a candidate that answers more questions by fabricating on two it used to refuse has a better average and is a worse system. So release comparison reads like a diff, not a scoreboard: what became better, what regressed, which failure class moved, what new tradeoff appeared. Answer recall up, but at whose expense? Refusals down: which causes went away, the over-cautious ones or the correct ones? Did retrieval improve while generation got less faithful, the two moving in opposite directions under one flat average? And sliced by question category and source, did one corner of the workload quietly regress while the aggregate held?
The shipping rule stays boring on purpose: the candidate ships only if it improves the cells that matter without buying them with regressions in fabrication, and the previous version stays deployable one step away, because an evaluation is a sample and production is the population.
Two hygiene rules keep the loop honest over time. The golden set rotates, because any set that gates promotion long enough gets overfit, deliberately or not; retired questions leave, adversarial newcomers enter, and scores are tracked across the seam. And the set grows from production: every real-world failure that reaches a human becomes a candidate question, which is how the evaluation stays adversarial after the authors have run out of imagination.
None of this is glamorous. Layered scoring, refusal taxonomies, full-set denominators, blind judges, failure-class diffs. But it is the difference between “our RAG scored well” and knowing, specifically, which decision your system gets wrong under pressure, which is the only thing an evaluation is for.