---
title: "RAG: Evaluation"
slug: "rag-eval"
category: "ai-agents"
tags: ["ai-agents", "rag", "evaluation", "metrics", "testing"]
status: "stable"
last_updated: 2026-05-14
summary: "Eval retrieval and generation separately. Recall@k for retrieval, faithfulness and answer relevance for generation, LLM-as-judge with skepticism."
related: ["[[rag]]", "[[evaluation]]", "[[rag-retrieval]]", "[[rag-reranking]]", "[[rag-citations]]"]
---

## Overview

RAG eval splits two questions. Did retrieval find the right context? Did generation use it correctly? A single end-to-end accuracy number hides which half is broken. The rules below build a golden set, score the two halves separately, and wire the suite into CI. For the general agent eval pattern, see [[ai-agents/evaluation]].

## Score retrieval and generation separately

Two metrics, two failure modes, two fixes.

- Retrieval: `recall@k` against gold chunk IDs. If `recall@5` is 0.4, no prompt edit saves you. See [[ai-agents/rag-chunking]], [[ai-agents/embeddings]], and [[ai-agents/rag-reranking]].
- Generation: `faithfulness` (did the answer stay in the retrieved context?) and `answer relevance` (does it respond to the question?). Both presume retrieval already worked.

Score independently, then track the joint number for release-blocking.

## Build a golden set of 50 to 200 Q&A pairs

A golden set is the contract for "is the new version better than the old one."

- 50 rows is the floor; 200 is enough for most production systems.
- Each row has: `question`, `gold_chunk_ids[]`, `gold_answer`, `slice_tag`.
- Half should be cases that already broke in production. The other half mixes easy, edge, adversarial, and time-sensitive.
- Treat the set as code. Version it, review changes in PRs, never edit a row to make a failing test pass.

Building the set takes a day; the system pays it back in the first week.

## Use recall@k at multiple k values

`recall@k` is the share of queries where the top-k retrieved chunks include at least one gold chunk. Track at multiple k.

- `recall@5`: what the prompt sees after reranking. The headline retrieval number.
- `recall@10`: reranker headroom.
- `recall@50`: candidate-pool ceiling. Low here means the dense or sparse stage is the bottleneck, not the reranker.

`recall@50` = 0.9 but `recall@5` = 0.5 says the reranker is the lever. Flat `recall@50` = 0.5 says the reranker is downstream of the real problem.

## Score generation with faithfulness and answer relevance

Once retrieval is right, the generation eval asks two questions.

- Faithfulness: is every factual claim in the answer supported by the retrieved context? Hallucinations show up here. Score with a judge model on a per-claim basis or with the RAGAS rubric.
- Answer relevance: does the answer respond to the question? An on-topic but evasive answer fails this. Score with an LLM judge on a 1 to 5 rubric, or by embedding similarity between question and answer.

Faithfulness without relevance ships safe, useless answers. Relevance without faithfulness ships confident hallucinations. The pair catches both.

## Use LLM-as-judge with skepticism

Judge models drift. Trust them only after they pass audit.

- Pick a stronger judge than the model under test.
- Score on a rubric, not a vibe: `"Does the answer cite the right chunk? yes/no."`
- Sample 30 to 50 rows per release for human rating. If judge-human agreement drops below 0.7, retire the judge or rewrite the rubric.

See [[ai-agents/evaluation]] for the broader LLM-as-judge harness.

## Track per-slice scores, not just the aggregate

An average score hides regressions on minority slices. A model 2 points better on average but 10 points worse on `adversarial` is a step backward.

- Slice tags: `easy`, `edge`, `adversarial`, `time-sensitive`, `multi-hop`, `keyword-heavy`.
- Dashboard columns: version, slice, `recall@5`, faithfulness, answer relevance, p95 latency, cost per query.
- Block merges on slice regressions, even when the aggregate moves up.

## Wire the suite into CI on every prompt or model change

Eval-driven iteration is the only way RAG gains stick. CI runs the suite; manual checks slip.

```text
prompt or model change → run eval suite → diff vs baseline →
block merge if any slice regresses by > N points or aggregate drops
```

- Cache embeddings and reranks where deterministic so the suite stays cheap. See [[ai-agents/embeddings]].
- Sample for every PR; run the full set nightly.
- A diff without an eval result is unreviewable.

## Inspect failures, do not just count them

The eval score points at problems; the failure log explains them. Read a sample of failed queries every release.

- Gold chunk not retrieved at all: retrieval failure.
- Gold chunk retrieved but not in the top-k: reranker failure.
- Chunk in the prompt but answer wrong: generation or citation failure. See [[ai-agents/rag-citations]].

Five minutes reading 10 failures beats an hour staring at the aggregate. The numbers tell you which knob; the failures tell you which turn.

## Related

- [[ai-agents/rag]]
- [[ai-agents/evaluation]]
- [[ai-agents/rag-retrieval]]
- [[ai-agents/rag-reranking]]
- [[ai-agents/rag-citations]]
- [[ai-agents/rag-chunking]]
- [[ai-agents/embeddings]]