LLM Best Practices

Observability

Last updated 4 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

Observability means answering questions about a running system without shipping a new build. Emit logs, metrics, and traces over OpenTelemetry (OTel), write structured logs from day one, and carry a trace ID through every request. Pick the vendor later; the instrumentation does not change.

Use all three signals for different questions

Logs say what happened (high-cardinality events for forensics). Metrics say how often and how slow (low-cardinality, cheap to dashboard). Traces say where the time went (one request across services). Wire all three: logs cannot show a latency trend, and metrics cannot explain one request.

Instrument with OpenTelemetry and OTLP

  • Traces, metrics, and logs are all stable in the OTel specification for the API and protocol. The metrics SDK is still marked mixed, and profiles are in development.
  • The OTel SDKs (Python, Node, Go, Rust, Java) send OTLP to an OTel Collector, which forwards to Datadog, Honeycomb, Grafana Cloud, Sentry, or a self-hosted stack. Switching backends is a Collector config change.
  • Use auto-instrumentation for HTTP servers, postgres clients, gRPC, and fastapi; add manual spans for business operations.
  • FastAPI 0.142 and later emits traces, metrics, and logs natively when the OpenTelemetry packages from fastapi[standard] are installed. Set OTEL_SERVICE_NAME and OTEL_EXPORTER_OTLP_ENDPOINT; turn parts off with the telemetry argument to FastAPI(...).
  • Prefer OTel over a vendor SDK as the primary integration, which would tie your wire format to one backend.

Know which semantic conventions are stable

Attribute names come from the semantic conventions (version 1.44.0 at the time of writing). HTTP and database conventions are stable; use http.request.method, not the old http.method. Instrumentations that predate the stable names emit both during migration when you set OTEL_SEMCONV_STABILITY_OPT_IN (for example http/dup or database/dup); drop the duplicates once dashboards use the new names. GenAI conventions have moved to their own repository; check their status before depending on those attribute names.

Ship structured logs from day one

Emit one JSON object per line with a stable schema, and serialize stack traces into a string field.

{"ts":"2026-10-01T12:00:00Z","level":"info","msg":"order.created","order_id":"ord_123","user_id":"usr_42","trace_id":"a1b2c3","span_id":"d4e5","duration_ms":42}
  • Top-level fields: ts, level, msg, service, env, trace_id, span_id. Add domain fields per event.
  • Use a logger that writes JSON natively: pino (Node), structlog (Python), zerolog (Go).

Propagate the trace ID on every hop

A trace is only useful if every log line and outbound call carries its ID.

  • Accept the W3C traceparent header on inbound HTTP; start a trace if it is missing.
  • Put the trace ID into the logger context (structlog.contextvars, pino child loggers).
  • Forward traceparent on outbound HTTP, gRPC, and queue messages. A trace that ends at a service boundary is half a trace.

Choose the right metric type

Use a counter for totals that only increase (http_requests_total), a gauge for values that move both ways (queue_depth), and a histogram for distributions (http_request_duration_seconds). Histograms give percentiles that aggregate across instances; client-side summaries do not. Keep label cardinality low: user_id and request_id as labels explode the metric store, so put them in logs and traces.

Build RED for services and USE for resources

  • RED for each service or route: Rate, Errors, Duration (p50, p95, p99). See fastapi for route-level shape.
  • USE for each host or hostinger-vps instance: Utilization, Saturation, Errors.

If a dashboard is neither RED nor USE, state which question it answers.

Send errors to a dedicated tracker and sample deliberately

Sentry, Rollbar, or GlitchTip group exceptions by fingerprint, deduplicate across replicas, and alert on the first occurrence in a release. Initialize the SDK once at process start and tag every error with release, env, and trace_id.

Drop DEBUG logs in production; use INFO for business events, not request chatter, WARN for recoverable problems, and ERROR for failures that need action. For traces, sample at the head to control volume and add tail-based sampling in the Collector so errored and slow requests are always kept. See general-principles for cost-aware telemetry.

Backlinks