Overview
Most production traffic does not need a frontier model, and sending all of it to one is the most common source of avoidable LLM spend. Routing assigns each request to a model tier based on difficulty; a fallback ladder retries a request on a stronger tier when the cheaper one fails a confidence check. Both depend on an eval suite that scores each tier per slice, so read evaluation first. This page is the routing half of cost-control.
Route by difficulty, not by default
Pick the smallest model that passes the eval for each request type. Run a router as the first hop.
- Triage with a small fast model (Haiku, GPT-5.6 Luna, Gemini Flash-Lite). It classifies the request.
- Routine work goes to a mid-tier model (Sonnet, GPT-5.6 Terra).
- Hard cases escalate to the frontier (Opus or Fable, GPT-5.6 Sol or GPT-6 Astra, Gemini Pro) on multi-step reasoning or low-confidence triage output.
For the Claude vs GPT vendor decision behind these tiers, see claude-vs-gpt.
Score each tier on the golden set. Promote a request only when the cheaper tier’s accuracy on that slice is below the bar.
Before building a multi-model cascade, measure the simpler alternative: one capable model at a lower effort setting. Prompt caches are scoped per model, so every extra tier forfeits cache reuse, and a cheaper request that needs more retries is not cheaper per completed task.
Build a fallback ladder
Order providers cheapest to most expensive and try them in series with a quality gate at each step.
1. Local model via Ollama for known-easy patterns (see [[ai-agents/ollama]]).
2. Cheap hosted API (Haiku, Gemini Flash-Lite, GPT-5.6 Luna) for routine tasks.
3. Frontier API (Opus, Gemini Pro, GPT-5.6 Sol) when steps 1 and 2 fail confidence checks.The ladder works only with a confidence signal at each rung. Without one, you pay for both calls.
Log every routing decision
Record the tier the router picked, whether the request escalated, and why (low triage confidence, a failed quality gate, a provider error) on every call. Without that log, a router that quietly sends most traffic to the frontier tier looks identical to one that works. Aggregate escalation rate per request type next to cost per task in the dashboards described in cost-control; a rising escalation rate is the first sign that a cheaper tier has stopped passing its slice.