Overview
Retrieval-augmented generation works when the retrieval step finds the right context. Most RAG failures are retrieval failures: the model writes confidently from whatever it is handed. This page is the map; each stage has its own page with the rules.
Skip RAG when the corpus fits in the prompt
Put a small, stable corpus directly in the prompt and use prompt caching. Anthropic’s contextual-retrieval guidance suggests this below roughly 200,000 tokens (about 500 pages). Larger context windows raise that ceiling, but you pay for every input token on every call unless the prefix is cached. Use RAG when the corpus exceeds the window, changes often, or must be filtered per user. See prompt-caching-strategies.
Follow the pipeline
| Stage | Rule | Page |
|---|---|---|
| Chunk | Split on structure, add per-chunk context and metadata | rag-chunking |
| Embed | Pick a model by eval, tag vectors with it, normalize | embeddings |
| Store | Choose by operations and filtering behavior | rag-vector-databases |
| Retrieve | Dense plus BM25 fused with RRF, metadata pre-filters | rag-retrieval |
| Rerank | Cross-encoder over 30 to 100 candidates | rag-reranking |
| Generate | Cite every claim; validate the pointers | rag-citations |
| Evaluate | Score retrieval and generation separately | rag-eval |
Diagnose in pipeline order: if the gold chunk is not retrieved, no prompt change helps.
Filter by freshness when the topic moves
Stale chunks poison answers. Attach last_updated to every chunk and apply a time filter when the query is time-sensitive (“latest”, “current”, “in 2026”).
- Hard filter: drop chunks older than N months for time-sensitive queries.
- Soft filter: boost newer chunks in the rerank score so older ones lose ties without being excluded.
Detect time sensitivity upstream of retrieval, then choose the filter mode.
Expose retrieval as a tool when one query is not enough
Let the model call a search tool and iterate when questions need several lookups or follow-ups. Return results as search_result blocks so citations come with them (see rag-citations). Cap the number of tool calls per question and log each query.
Cache embeddings and retrievals
- Embeddings: key on model, dimension,
input_type, and text hash. See embeddings-cost-control. - Retrievals: key on the normalized query text and filters, with a short TTL, and include the corpus version so re-indexing invalidates entries.
- Answers: reuse by meaning only after measuring false positives; see embeddings-semantic-cache.
Treat retrieved text as untrusted
A chunk is data, not instructions. Put chunks in a delimited block, tell the model not to follow instructions inside them, and enforce tenant filters in the retrieval query rather than in the prompt. See prompt-injection-defense.