LLM Best Practices

Model Routing and Fallback Ladders

4 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

Most production traffic does not need a frontier model. Routing assigns each request to a model tier by difficulty, and a fallback ladder retries on a stronger tier when the cheaper one fails a confidence check. Both depend on an eval suite that scores each tier per slice, so read evaluation first. This page is the routing half of cost-control.

Route by difficulty, not by default

Give each request type the smallest model that passes the eval for that slice, and use a small model as the router.

Claude tier (API ID)Input / output per million tokensContextUse for
Haiku 4.5 (claude-haiku-4-5)5200KTriage, classification, extraction
Sonnet 5.5 (claude-sonnet-5-5)101MRoutine agent and coding work
Opus 5.5 (claude-opus-5-5)201MMulti-step reasoning, hard cases
Fable 5.1 (claude-fable-5-1)501MThe hardest tasks; most capable

Prices are list prices as of 2026-10-01. Other vendors sell the same three tiers: OpenAI’s current API models include gpt-6-luna (lowest cost), gpt-6.1-sol, and gpt-6-astra (flagship), and Google’s include gemini-3.8-flash (latest stable Flash) and gemini-3.1-pro-preview (top Pro). Names and prices change quickly, so check the vendor’s pricing page, and see claude-vs-gpt for the vendor decision.

Check the ratio before building a router: Haiku 4.5 is half the per-token price of Sonnet 5.5, so a router wins only if it sends most traffic down a tier without adding retries or failed tasks. Promote a request type to a stronger tier only when the cheaper tier scores below the bar on that slice.

Try one model at a lower effort first

Before a multi-model cascade, measure one capable model at a lower effort setting. Prompt caches are scoped per model, so every extra tier forfeits cache reuse, and a cheaper request that needs more retries is not cheaper per completed task.

Normalize request parameters per tier

A router that sends one request body to every tier breaks on API differences.

  • Sampling parameters: Opus 4.7 and later (including Opus 5 and 5.5), Sonnet 5 and 5.5, and Fable 5 and 5.1 accept only the defaults (temperature 1.0, top_p 0.99 or higher) and return 400 for other values and any top_k. Sonnet and Opus 4.6 and Haiku 4.5 still accept them.
  • Thinking uses thinking: {"type": "adaptive"} plus output_config.effort on current models, but Haiku 4.5 still needs budget_tokens, which is removed (HTTP 400) on Opus 4.7 and later.
  • Forced tool_choice (any or tool) returns 400 on Opus 5.5, Sonnet 5.5, and Fable 5.1; use auto with strict: true tools.
  • Assistant-message prefill returns 400 on the 4.6 and later families, including all 5.x models; use structured outputs or system instructions.

Keep a per-tier parameter profile in the router and test every tier in CI.

Build a fallback ladder

Order models cheapest to most expensive and try them in series with a quality gate at each rung.

1. Local model via Ollama for known-easy patterns (see ai-agents/ollama).
2. Small hosted model (Haiku 4.5) for routine tasks.
3. Frontier model (Opus 5.5 or Fable 5.1) when rungs 1 and 2 fail their gate.

The ladder needs a confidence signal at each rung (a schema check, a test run, a judge score); without one you pay for both calls. Keep a provider failover separate from the quality ladder, so an outage does not look like a quality failure.

Log every routing decision

Record the tier picked, whether the request escalated, and why (low triage confidence, failed gate, provider error). Without that log, a router quietly sending most traffic to the frontier tier looks identical to one that works. Aggregate escalation rate per request type next to cost per task; a rising rate is the first sign that a cheaper tier has stopped passing its slice.

Backlinks