LLM Best Practices

LLM-as-judge

Last updated 2 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

LLM-as-judge uses a language model to score another model’s output against a rubric. The judge reads the task, the candidate answer, and optionally a reference answer, then returns a score and a rationale. It lets evaluation scale past hand-grading.

Judges have known biases: position bias (favoring the first option shown), verbosity bias (favoring longer answers), and self-preference (rating output from the same model family higher). Control them by swapping candidate order and averaging, writing a specific rubric with anchored score descriptions, requesting the rationale before the score, and using a different model family as judge where possible.

Calibrate before trusting trends. Grade a sample by hand from the golden-set, measure agreement between judge and humans (rank correlation or Cohen’s kappa), and recheck whenever the judge model or rubric changes. Skip the judge for high-stakes items where one wrong grade costs more than human review.

Example

A RAG eval runs 500 questions. A judge rates each answer from 1 to 5 for groundedness against the retrieved chunks; the mean is the run’s groundedness metric and the rationales help debug regressions.

Backlinks