---
title: "Distillation"
slug: "distillation"
category: "glossary"
tags: ["glossary", "ai-agents", "distillation", "training", "model", "llm", "efficiency"]
status: "stable"
last_updated: 2026-08-14
summary: "Knowledge distillation trains a small student model to mimic a larger teacher, producing compact weights that approach teacher performance at lower cost."
related:
  [
    "[[glossary/fine-tuning]]",
    "[[glossary/evaluation-harness]]",
    "[[glossary/golden-set]]",
    "[[glossary/completion]]",
    "[[prompt-engineering/prompt-design]]",
  ]
---

## Overview

This page is the atomic definition. The prompting and model-selection deep-dive lives at [[prompt-engineering/prompt-design]]. Knowledge distillation trains a smaller student model to reproduce a larger teacher model's outputs or logits, yielding a compact model that approaches teacher quality at lower inference cost.

## Definition

Knowledge distillation is a training technique where a smaller student model is trained to match the behavior of a larger teacher model. The teacher generates outputs (soft labels, logit distributions, or full completions) on a dataset; the student is trained to reproduce them. The result is a compact model that retains most of the teacher's quality on the target task at a fraction of the inference cost.

Two main forms:

1. Output distillation (black-box): the teacher generates text completions on a prompt set; the student fine-tunes on those completions. This works with API-only access to the teacher. The student learns to match the teacher's outputs.

2. Logit distillation (white-box): the student minimizes the KL divergence between its token probability distribution and the teacher's. Requires access to the teacher's internal logits. More data-efficient than output distillation.

Distillation differs from [[glossary/fine-tuning|fine-tuning]] in that the training signal is machine-generated from a stronger model rather than human-labeled. It is also distinct from quantization (which compresses weights without retraining).

## When it applies

Use distillation when you need a smaller, faster, cheaper model for a specific task and a strong teacher model is available. Collect teacher outputs on a representative dataset (1,000 to 100,000+ examples). Evaluate the student against the teacher on a held-out test set using an [[glossary/evaluation-harness|evaluation harness]]. Distillation works best when the task is well-defined and the teacher's outputs are high quality.

Do not expect the student to generalize beyond the teacher's distribution; distillation is task-specific.

## Example

```python
import anthropic
client = anthropic.Anthropic()

prompts = load_training_prompts()  # your task dataset
teacher_examples = []

for prompt in prompts:
    response = client.messages.create(
        model="claude-opus-5",  # teacher
        max_tokens=512,
        messages=[{"role": "user", "content": prompt}]
    )
    teacher_examples.append({
        "prompt": prompt,
        "completion": response.content[0].text
    })

save_jsonl(teacher_examples, "distillation_train.jsonl")
```

## Related concepts

- [[glossary/fine-tuning]] - distillation uses machine-generated labels; fine-tuning uses human-labeled data. Both update model weights.
- [[glossary/evaluation-harness]] - measure whether the student matches the teacher on the target task.
- [[glossary/golden-set]] - use a golden set to cap acceptable quality degradation from teacher to student.
- [[glossary/completion]] - distillation uses completions from the teacher as training targets.
- [[prompt-engineering/prompt-design]] - prompting the teacher to produce high-quality training examples is critical.

## Citing this term

> See [[glossary/distillation|Distillation]] (llmbestpractices.com/glossary/distillation).

## Related

- [[glossary/fine-tuning]]
- [[glossary/evaluation-harness]]
- [[glossary/golden-set]]
- [[glossary/completion]]
- [[prompt-engineering/prompt-design]]