LLM Best Practices

LLM observability and monitoring guide

Last updated 2 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

LLM observability instruments an LLM system so you can answer what happened on any request and whether quality is drifting. It extends logs, metrics, and traces with tokens, cost, prompt and model versions, tool-call spans, and quality scores. Without it a nondeterministic failure cannot be reproduced. Pillar: llmops-best-practices; measurement methods: llm-evaluation-in-production; general practice: observability.

Emit one trace per request with a correlation ID

Span the whole path: the assembled prompt, the model ID and prompt version, retrieved chunks, each tool call and its result, the response, and the stop reason. For agents, add a span per loop iteration to see where the run went wrong. Everything else is aggregation over traces (reliable-agents-in-production, agent-loop).

Track the metrics classic APM misses

  • Input and output tokens, plus cost, per request, model, and feature.
  • Time to first token and total latency; time to first token matters most when streaming.
  • Prompt-cache hit rate (cost-control).
  • Tool-call count, tool-error rate, and loop-cap hits per request.

Monitor quality, not just uptime

A system can be fully up and quietly wrong. Sample live traffic for thumbs up or down, edit and regenerate rates, refusal rate, escalation rate, and faithfulness or hallucination on retrieval-grounded answers. Feed graded samples back into the golden set (llm-evaluation-in-production).

Log prompts and outputs with privacy controls

Store inputs and outputs for debugging, but redact secrets and PII at capture (not after storage), set retention limits, and respect user consent. Raw conversation logs without controls are a data-protection liability. Keep citation provenance in the trace so a disputed answer can be audited (rag-citations).

Alert on failures that hide

Alert on a spike in loop-cap hits, a rising tool-error rate, a cost-per-request jump, a drop in thumbs-up rate, and a latency regression after a deploy. Tag every alert with prompt and model version so a regression maps to a change.

Verify the setup

  • A failing request’s full trace is retrievable by correlation ID.
  • Dashboards break down tokens, cost, and latency by model and feature.
  • PII redaction runs at capture.
  • Deploying a deliberately worse prompt to staging triggers a quality alert.

Backlinks