Overview
Agent evaluation produces a number you can trust for “is this version better than the last”. Without it every prompt change is a guess and every regression ships quietly.
Build the golden set before tuning
Start small with cases that mirror real traffic; Anthropic’s research team began with about 20 queries and grew from there. The set is the contract.
- Each row has an input, the expected output or an accept-list, and a slice tag (
easy,edge,adversarial,time-sensitive). - Include the long tail: seed it with cases that already broke in production.
- Treat it as code. Version it, review changes in PRs, and never edit a row to make a failing test pass.
- Prefer many cases you can grade automatically over few hand-graded ones; grow the set as failures appear.
Grade with code first, then an LLM judge
Use exact match or code checks wherever the answer is checkable (a test suite, a schema, a string). Use an LLM judge for open-ended output, then audit it.
- Use a different model from the one that generated the output.
- Write a detailed rubric with a discrete verdict: “Does the answer cite the right source? yes or no.” Let the judge reason before the verdict, then discard the reasoning.
- Compare judge and human labels on a fresh sample each release, using simple agreement or Cohen’s kappa against a bar you set in advance. Retire or rewrite a judge that falls below it. Judges drift too; see llm-as-judge and structured-output for keeping verdicts parseable.
Record a baseline before any change
Run the current production version on the full golden set and store its score; that is the bar. Ship only if the new version beats the baseline in aggregate and does not regress any named slice beyond a margin you chose in advance. Two runs of the same prompt differ by a few points, so treat results inside that noise as a tie and ship the simpler prompt, cheaper model, or faster path.
Split retrieval eval from generation eval in RAG
A wrong answer can mean retrieval missed the chunk or generation ignored it. Score recall@k against gold chunk IDs separately from answer correctness given the retrieved context. If retrieval recall is low, no prompt edit will help. See rag-eval.
Use pass@k when many outputs are valid
For code, queries, or plans with several acceptable answers, run k samples and count the task passed if any is correct. pass@1 is the experience a user sees; the gap to pass@5 is the headroom a reranker or retry could buy.
Run the suite on every change and watch per-slice scores
Wire the suite into CI and block merges on a regression in any named slice or in the aggregate. A diff without an eval result is not reviewable. Publish version, slice, accuracy, judge agreement, p50 and p95 latency, and cost per task on one dashboard: a model 2 points better on average but much worse on adversarial is a step backward.
Score cost and latency with quality
A version slightly more accurate at several times the cost usually loses. Cost per task and p95 latency belong in the eval; see cost-control and the token budgets in multi-agent.
Refuse to ship on vibes
“It feels smarter” is not a release criterion; it accumulates silent regressions nobody can localize. If you cannot show the eval diff, do not ship. For the acceptance-criteria version of this rule see claude-code, and for the code analogue general-principles.