LLM Best Practices

Structured Output

4 min read

View markdown source

Overview

Anything a program will consume must be JSON validated against a schema. Prose is for humans; downstream code needs typed, predictable shape. Modern APIs from Anthropic, OpenAI, and Google all support schema-constrained output. Use it. Never regex a JSON object out of free-form prose; you will spend a year debugging the edge cases.

Define the schema first

Write the schema before the prompt. The schema is the contract; the prompt is the implementation detail that satisfies it. Contract-first design surfaces ambiguity early.

{
  "type": "object",
  "required": ["verdict", "reasons"],
  "additionalProperties": false,
  "properties": {
    "verdict": { "enum": ["approve", "request_changes", "block"] },
    "reasons": {
      "type": "array",
      "minItems": 1,
      "maxItems": 5,
      "items": { "type": "string", "maxLength": 200 }
    }
  }
}

Bounded enums, bounded arrays, bounded strings. Every field that does not have a max is a future incident.

Use tool use or strict mode, not prose-with-JSON

Every major provider has a schema-constrained mode. Use it.

  • Anthropic: tool use with an input schema; the model can only emit matching JSON.
  • OpenAI: response_format: { type: "json_schema", strict: true } or function calling with strict: true.
  • Google Gemini: responseMimeType: "application/json" plus responseSchema.

“Return JSON in this shape” in a system prompt is a hope, not a contract. Strict modes enforce it at the decoder.

Validate at the boundary

Treat the model like an untrusted external service. Validate every response against the schema before any downstream code touches it.

  • Python: pydantic.BaseModel.model_validate(payload). The instructor library wraps this around the API call.
  • TypeScript: zod.parse(payload) or ajv with the same JSON Schema you sent the model. See typescript-runtime-validation.
  • On validation failure: retry with the validation error in the next user message, or fall back to a smaller model.

Validation is cheap; cleanup of a bad row written into Postgres is not. See postgres.

Never parse JSON from prose with regex

A regex that pulls {...} out of a model response will fail on nested braces, code fences, escaped quotes, multi-line strings, and the day the model emits two JSON objects. The structured-output APIs exist for a reason.

  • If you must extract JSON from prose, parse a JSON balanced substring with a stack-based parser, not regex.
  • If you must do that on a hot path, switch to tool-use mode. The work is already done for you.

The cost of switching to tool-use is one afternoon. The cost of regex-parsed JSON in production is six months of JSONDecodeError alerts.

Strict mode catches type drift

Strict mode rejects unknown properties, type mismatches, and unbounded arrays at the decoder. Without it, the model invents fields when it gets uncertain.

  • Anthropic tool use: define the input schema with additionalProperties: false. Unknown keys are not generated.
  • OpenAI strict mode: every field must be declared; additionalProperties defaults to false. The model cannot add a "note" field on a whim.

Loose schemas drift. A "metadata": {} open object becomes a junk drawer. Constrain or do not include.

Combine reasoning with a structured answer

When the task benefits from chain-of-thought, put the reasoning in one field and the answer in another. The schema separates concerns; the caller reads the answer; the log keeps the reasoning. See chain-of-thought.

{
  "type": "object",
  "required": ["reasoning", "answer"],
  "properties": {
    "reasoning": { "type": "string", "maxLength": 2000 },
    "answer": { "type": "string", "maxLength": 200 }
  }
}

Without the structure, the reasoning leaks into the user-facing answer or vanishes from the logs entirely.

Pydantic plus instructor is the Python default

For Python services, the pairing of pydantic for the schema and instructor for the API client gives you typed models, retries on validation failure, and structured output across providers with one interface.

import instructor
from pydantic import BaseModel, Field
from anthropic import Anthropic
 
class Review(BaseModel):
    verdict: str = Field(pattern="^(approve|request_changes|block)$")
    reasons: list[str] = Field(min_length=1, max_length=5)
 
client = instructor.from_anthropic(Anthropic())
result = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    response_model=Review,
    messages=[{"role": "user", "content": "Review PR #401"}],
)

The Pydantic model is the schema and the type at the same time. See fastapi for the same pattern at the API boundary.

Version the schema like an API

A schema is a public interface. Treat schema changes like API breaks.

  • Add fields with safe defaults; never repurpose a field.
  • Rename only via additive deprecation: new field, parallel run, retire old field.
  • Tag the schema version in logs so a downstream parser can choose the right decoder.

A schema change without a version bump is the same incident as an unannounced REST breaking change.

Graph View

Backlinks