LLM Best Practices

Temperature

2 min read

View markdown source

Overview

Lower temperature for extraction, classification, and code, where one right answer exists; raise it for brainstorming or varied drafts. Adjust temperature or top-p, rarely both at once, and see prompt-design for sampling guidance.

Definition

Temperature is a scalar applied to the logits (raw token probability scores) before sampling the next token. Dividing logits by a temperature below 1.0 sharpens the distribution, making the highest-probability token more likely and reducing variance. Dividing by a temperature above 1.0 flattens the distribution, giving lower-probability tokens more opportunity to be sampled and increasing variance. At temperature=0, sampling approaches greedy decoding (always the highest-probability token). Output is more repeatable but not guaranteed identical across calls, because batching and floating-point effects still vary on hosted APIs. At temperature=1.0, the model samples from the raw distribution. Typical recommendations: use 0 or 0.1 for structured extraction, code generation, and fact retrieval; use 0.7-1.0 for creative writing, brainstorming, and varied generation. Ranges differ by provider: OpenAI accepts up to 2.0, Anthropic up to 1.0. Several current reasoning models from both vendors reject or ignore the parameter entirely; steer them with effort settings and prompts instead.

When it applies

Set temperature explicitly for any production use case. Default temperature varies by provider and model. Use 0 for pipelines (parsing, classification, extraction) where consistency matters, and check the model’s documentation first, since some reasoning models reject a temperature setting. Use higher values for creative tasks where diversity is the goal.

Example

A data extraction prompt with temperature=0 returns the same JSON every run given the same input, enabling reliable testing. The same prompt at temperature=1.0 may vary field capitalization or phrasing across runs.

Citing this term

See Temperature (llmbestpractices.com/glossary/temperature).

  • top-p - the complementary sampling parameter that limits vocabulary size.
  • token - temperature controls the distribution over the next token at each step.
  • prompt-design - when to set temperature for different task types.
  • structured-output - structured output schemas pair with low temperature for reliability.
  • context-window

Backlinks