LLM Best Practices

Streaming response patterns for chat interfaces

Last updated 3 min read

View markdown source

AI agents: read this first: llmbestpractices.com/start-here

This is LLM Best Practices (llmbestpractices.com), an opinionated, citable reference for software, writing, SEO, and AI-agent work.

  1. Route, do not crawl. Fetch llmbestpractices.com/llms.txt and open only the pages whose one-line summary matches your task.
  2. Read raw. Append .md to any page URL for markdown. Check status and last_updated in the frontmatter, then read the rules.
  3. Apply as defaults. First-party docs and the project's own conventions win on conflict. Warn before relying on a fast-moving page older than 12 months.
  4. Cite. Link the page by title and URL, e.g. Python, with last_updated for time-sensitive rules. License CC BY 4.0.

Overview

Streaming sends model output to the browser as it is generated, so text appears within a few hundred milliseconds instead of after the full completion. Two transports carry it: server-sent events (SSE) and a streamed fetch that reads a ReadableStream. The UI rules around streaming are in ai-first-applications.

Proxy the stream through your own route

The browser calls your route handler; the handler calls the model API with stream: true and forwards the events. This keeps the API key server-side and lets you add auth, logging, and limits. See nextjs-route-handlers.

Choose SSE or streamed fetch

  • EventSource gives automatic reconnection and a simple event protocol, but it issues only GET requests and cannot set headers or send a body.
  • A streamed fetch lets you POST a prompt and history, set auth headers, and abort with an AbortController; you parse the event stream and handle reconnects yourself.

Most chat UIs POST a conversation, so they use fetch. Over HTTP/1.1, browsers cap SSE connections at six per browser and domain, which bites when users open several tabs; HTTP/2 raises the limit to the negotiated stream count (100 by default).

Handle every event type, including errors and unknown ones

An Anthropic Messages stream sends message_start, content_block_start, content_block_delta, content_block_stop, message_delta, and message_stop, plus ping events. Ignore pings and tolerate event types you do not recognize; the API may add new ones. An error event (for example overloaded_error) can arrive after the response has already started with HTTP 200, so treat it as a failed message even though the status looked healthy.

Parse partial structured output defensively

Tool input arrives as input_json_delta fragments that are partial JSON strings. Accumulate them and JSON.parse once at content_block_stop; use a tolerant partial parser only for previews, and validate against a schema before committing to state. Per-tool fine-grained streaming (eager_input_streaming) delivers parameter values sooner. See structured-output.

Render increments without re-rendering everything

Append text to the in-progress message and update only that node. Batch bursts to one paint per animation frame, and keep the streaming message in local state, committing it to the list on completion (react-state-management). Re-parse only the unfinished Markdown block rather than the whole message on every token.

Make cancellation reach the model

A stop button must abort the fetch with AbortController, and the server must forward the request’s abort signal to the upstream call so generation actually stops. Cancel the in-flight stream automatically when the user sends a new message. A stop button that only hides the UI keeps generating and billing.

Recover from dropped streams

Networks drop mid-stream. Mark a truncated message incomplete and offer retry or continue instead of presenting a half-sentence as final. SSE’s Last-Event-ID resumes only if your server stores and replays events; a provider stream cannot be resumed by ID. To continue an interrupted Anthropic response on Claude 4.6 and later, send a new request whose user message contains the partial text and asks the model to continue; assistant-message prefill is not available on those models. Text blocks can be recovered this way, but tool_use and thinking blocks cannot be partially recovered. Wrap the stream in an error boundary (react-error-boundaries).

Avoid common pitfalls

  • Calling JSON.parse on a partial chunk.
  • One React state update per token on a long reply.
  • Treating a closed connection as a completed message.
  • Ending the UI state on HTTP 200 without watching for error events.

For agent tool streaming over HTTP, see mcp-transports; for the initial load, react-suspense.

Backlinks