LLM Best Practices

Cost Control

Last updated 4 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

Agent cost falls through a stack of habits rather than one switch: a cached prefix, the right model tier, batch processing for offline work, bounded context, and caps with metrics. Each assumes an eval suite exists; cheap and wrong is still wrong, so see evaluation first. Send each request to the smallest model that passes the eval; tiers, prices, and the fallback ladder are in model-routing.

Measure cost per task, not cost per token

Sum every token spent to deliver one user-visible outcome: retries, judge calls, tool calls, planner overhead. Track it per slice (easy, hard, adversarial) so a price change cannot hide behind a healthy average. A cheaper model that loops three times costs more.

Cache the stable prefix

On the Claude API a prompt is matched as a prefix in render order: tools, then system, then messages.

  • Put durable content first (tool definitions, system prompt, schema, few-shot examples) and per-turn input last. Keep the prefix byte-stable: no timestamps or request IDs in the system prompt, and a deterministic tool order.
  • Mark breakpoints with cache_control (at most 4 explicit), or set one top-level cache_control for automatic caching. Entries default to a 5-minute TTL; "ttl": "1h" extends it. A 5-minute write costs 1.25 times base input, a 1-hour write 2 times, and a read 0.1 times (0.05 on Opus 5.5, 0.025 on Fable 5.1). Use 1 hour when requests arrive less often than every 5 minutes.
  • Prefixes below the model’s minimum (between 512 and 4,096 tokens depending on model) are not cached, and no error is returned.
  • An entry is readable only after the first response begins, so concurrent requests miss; wait for the first response before fanning out.
  • Changing tool definitions invalidates the whole cache. Toggling tool_choice or adding or removing images invalidates cached messages. Changing the thinking configuration or effort between requests also breaks cache breakpoints on current models, so pick one setting per conversation and steer individual turns with a prompt instead.
  • Caches are isolated per workspace on the Claude API, Claude Platform on AWS, and Microsoft Foundry, and per organization on Bedrock and Google Cloud.

Track cache_read_input_tokens against cache_creation_input_tokens in usage. OpenAI and Gemini cache implicitly by default; check each provider’s minimum length and discount. See prompt-caching-strategies for prefix ordering.

Control thinking spend

On current Claude models use thinking: {"type": "adaptive"} with output_config: {"effort": "low" | "medium" | "high" | "xhigh" | "max"}; the default is high on most models and medium on Opus 5.5. Thinking tokens are billed as output tokens, and usage.output_tokens_details.thinking_tokens shows how many. Lower effort is the first lever for over-thinking. max_tokens is the hard cap on thinking plus answer per request, so size it for both, and remember each request in a tool loop has its own cap.

Use batch APIs for anything not interactive

The Anthropic Message Batches API and the OpenAI Batch API each take 50 percent off in exchange for asynchronous results; OpenAI’s window is 24 hours. Put eval suites, embedding backfills, nightly summaries, and bulk classification on batch, and keep realtime for user-facing turns.

Trim context and cache deterministic tool results

  • Drop stale tool results once the next step is planned, and replace old turns with a short summary.
  • In RAG, send a few reranked chunks, not dozens; see rag-retrieval.
  • Memoize tools that return the same output for the same input, keyed on (tool_name, serialized_args) with a TTL matched to the data: minutes for web fetches, hours for slow-changing rows. Embedding calls key on (model_id, sha256(text)) and can live near-permanently; see embeddings.

Cap spend and observe it per session

Every entry point needs a budget: a per-task token ceiling that returns a partial result and a budget_exceeded flag, a per-user daily cap that throttles or downgrades, and an organization cap with alerts before it is hit. This is the dollar version of the iteration caps in multi-agent.

Billing data lags, so emit input_tokens, output_tokens, cache_read_input_tokens, cache_creation_input_tokens, and computed cost_usd per call and aggregate by session and task. Chart cost per task by slice and cache hit rate, and alert when a task type’s cost doubles week over week; a prompt regression that doubles the bill is invisible without per-session attribution.

Backlinks