Evaluating RAG Systems

Evaluate retrieval and generation separately

A RAG pipeline fails in two places: it retrieves the wrong chunks, or it generates a bad answer from good chunks. A single end-to-end score cannot tell these apart, and the fix for each is completely different. Every evaluation setup should therefore produce one number for retrieval quality and one for generation quality per query.

Start with a labeled question set, often called a golden set. Each entry needs a realistic question, the reference answer, and the ids of the chunks that support it. Fifty to a hundred questions across the corpus is enough to start; include paraphrases of the same question, keyword-style queries, multi-hop questions that need two chunks, and a few unanswerable questions where the correct behavior is to say the information is not available.

Retrieval metrics

Retrieval metrics are computed against the labeled chunk ids and need no language model. Recall at k is the fraction of the relevant chunks that appear in the top k results. Precision at k is the fraction of the top k results that are relevant. Mean reciprocal rank (MRR) is the average of one over the rank of the first relevant result, which rewards putting the right chunk first.

```python

def recall_at_k(retrieved_ids, relevant_ids, k):

retrieved = set(retrieved_ids[:k])

relevant = set(relevant_ids)

return len(retrieved & relevant) / len(relevant) if relevant else 0.0

def mrr(retrieved_ids, relevant_ids):

for rank, chunk_id in enumerate(retrieved_ids, start=1):

if chunk_id in relevant_ids:

return 1.0 / rank

return 0.0

```

For RAG, recall at k matters more than precision at k, because a reranker or the model itself can ignore extra chunks but cannot use chunks that were never retrieved. Track recall at the k you actually send to the model, and at the larger k you send to the reranker if you use one. When recall is low, the levers are chunk size, the embedding model, hybrid search weights, and the first-pass k.

Generation metrics

Generation metrics judge the answer against the retrieved context and the question. Faithfulness asks whether every factual claim in the answer is supported by the retrieved chunks; it is the direct measure of hallucination. Answer relevancy asks whether the answer addresses the question asked, which catches verbose or off-topic responses that are technically grounded.

These are graded by an LLM judge in modern pipelines, because string matching against a reference answer penalizes correct paraphrases. The judge setup needs care: use a different model than the generator when possible, pin the judge model version so scores are comparable across runs, and spot-check a sample of judgments by hand on every run. Judge bias is real, and an unvalidated judge just moves the unreliability one layer up.

RAGAS-style evaluation

RAGAS (open source at github.com/explodinggradients/ragas, paper arXiv:2309.15217) packages this into four reference-light metrics: faithfulness and answer relevancy for the generation half, context precision and context recall for the retrieval half. Context precision measures whether the retrieved chunks are free of noise, and context recall measures whether retrieval covered everything the reference answer needs.

The diagnostic value is the mapping from metric to fix. Low faithfulness points at the prompt, the temperature, or the generator model. Low answer relevancy points at noisy context or weak instruction following. Low context precision points at top k or the reranker threshold. Low context recall points at chunk size, the embedding model, or hybrid search. Run the suite on every pipeline change and gate releases on the scores not regressing.

Human evaluation

Automated metrics miss what humans notice: tone, over-hedging, citation accuracy, and whether the answer is actually useful. A lightweight human review loop is still the highest-signal evaluation for the questions that matter most.

The workable format is side-by-side comparison of two pipeline variants on the same questions, with raters picking the better answer and noting why. Twenty to thirty questions per round is enough to catch regressions that metrics miss. Keep a standing set of adversarial questions drawn from real user queries that previously failed, and re-run them after every change to chunking, retrieval, or prompts.

Failure modes to watch for

Lost in the middle is an empirical property of long-context models, documented by Liu et al. (arXiv:2307.03172): models use information at the start and end of a long context well and degrade on information buried in the middle, even models built for long contexts. In RAG this means stuffing more chunks into the prompt can lower accuracy. Order retrieved chunks by relevance, keep the prompt to the chunks that matter, and retest accuracy as k grows.

Retrieved junk is the most common production failure: the answer is in the corpus, but the top k is full of near-duplicate or tangentially related chunks and the true chunk ranks below the cutoff. Symptoms are low context precision with acceptable faithfulness. Fixes are raising the reranker threshold, tightening metadata filters, and deduplicating near-identical chunks at index time.

Chunk-boundary splits hide facts that span two chunks; neither chunk alone matches the query well. Overlap and parent-document retrieval address this directly. Contradictory sources are a corpus problem: two documents disagree and the model picks one silently. Deduplication, date filtering, and prompting the model to surface disagreement when sources conflict are the available mitigations.

Putting it in production

Offline evaluation on the golden set runs on every change. Online, log the question, the retrieved chunk ids, and the answer for a sample of traffic, and compute faithfulness and answer relevancy continuously with the same judge setup. Alert on drops, and feed failing queries back into the golden set. The evaluation set is a living artifact: every production failure that was not covered by a test case is a missing test case.