---
title: "LLM-as-judge"
slug: "llm-as-judge"
category: "glossary"
tags: ["glossary", "ai-agents", "evaluation", "llm-as-judge", "metrics"]
status: "stable"
last_updated: 2026-05-14
summary: "LLM-as-judge is the practice of using a language model to score the outputs of another model against a rubric, used to scale evaluation past hand-grading."
related:
  [
    "[[ai-agents/evaluation]]",
    "[[glossary/golden-set]]",
    "[[ai-agents/rag-eval]]",
    "[[glossary/hallucination]]",
    "[[prompt-engineering/prompt-design]]",
    "[[ai-agents/evaluation]]",
  ]
---

## Overview

This page is the atomic definition. The deep-dive lives at [[ai-agents/evaluation]].

## Definition

LLM-as-judge is the practice of using a language model to score the outputs of another model against a rubric. The judge model reads the task, the candidate response, and (optionally) a reference answer, then returns a score plus rationale. Used to scale evaluation past hand-grading. Judge prompts must be calibrated: known failure modes include positional bias (preferring the first option presented), verbosity bias (preferring longer answers), and self-preference (a model rating its own outputs higher). Pair LLM-as-judge with a hand-graded golden set to verify the judge agrees with human raters at acceptable correlation (often Pearson 0.6+).

## When it applies

Use LLM-as-judge for large-scale eval runs (hundreds to thousands of items) where hand-grading is too expensive. Skip it for high-stakes evals where the cost of a misgraded item outweighs the cost of human review.

## Example

A RAG eval runs 500 test questions through the system. A judge model rates each answer 1-5 for groundedness against the retrieved chunks. The aggregated score is the run's groundedness metric; the rationale lets engineers debug regressions.

## Related concepts

- [[ai-agents/evaluation]] - the deep-dive with rubric and calibration patterns.
- [[glossary/golden-set]] - the hand-labeled ground truth that calibrates the judge.
- [[ai-agents/rag-eval]] - RAG-specific eval patterns where LLM-as-judge is common.
- [[glossary/hallucination]] - the most common failure mode the judge looks for.
- [[prompt-engineering/prompt-design]] - the discipline that produces a calibrated judge prompt.

## Citing this term

> See [[glossary/llm-as-judge|LLM-as-judge]] (llmbestpractices.com/glossary/llm-as-judge).

## Related

- [[ai-agents/evaluation]]
- [[ai-agents/rag-eval]]
- [[glossary/golden-set]]
- [[glossary/hallucination]]
- [[prompt-engineering/prompt-design]]