---
title: "RAG Best Practices"
slug: "rag"
category: "ai-agents"
tags: ["ai-agents", "rag", "reranking"]
status: "stable"
last_updated: 2026-05-29
summary: "Chunking, retrieval, reranking, evaluation, and citation patterns that ship reliable RAG systems."
related: ["[[ai-agents/embeddings]]", "[[backend/chromadb]]", "[[prompt-engineering/prompt-design]]", "[[ai-agents/mcp-servers]]", "[[ai-agents/rag-vector-databases]]", "[[ai-agents/rag-citations]]"]
---

## Overview

Retrieval-augmented generation works when the retrieval step actually finds the right context. Most RAG failures are retrieval failures, not generation failures; the model writes confidently from whatever you hand it. The rules below cover [[ai-agents/rag-chunking|chunking]], [[ai-agents/rag-retrieval|retrieval]], [[ai-agents/rag-reranking|reranking]], and [[ai-agents/rag-eval|evaluation]]. For the vector store, see [[backend/chromadb]]. For embedding models, see [[ai-agents/embeddings]].

## Chunk on semantic boundaries, not fixed character counts

Split documents on headings and paragraph breaks first, then enforce a token budget. A 500-token chunk that starts mid-sentence is worse than a 700-token chunk that respects the section.

- Target 200 to 800 tokens per chunk. Below 200, you lose context; above 800, retrieval gets fuzzy.
- Split on `#`, `##`, blank lines, then code-fence boundaries.
- Add a 50-token overlap between adjacent chunks so a fact split across a boundary stays retrievable.

Attach metadata to every chunk: `source_url`, `title`, `heading_path`, `last_updated`. The metadata is half the system.

## Combine dense embeddings with metadata filters

Pure semantic search is not enough when the user is specific. "Postgres 17 release notes" needs both vector similarity and a filter on `version = "17"`.

```python
results = collection.query(
    query_embeddings=[query_vec],
    where={"category": "backend", "last_updated": {"$gte": "2026-01-01"}},
    n_results=50,
)
```

Use BM25 as a parallel retriever when the query has rare keywords (error codes, identifiers, exact phrases). Merge with reciprocal rank fusion. Pure dense retrieval misses `EADDRINUSE`; BM25 catches it.

## Rerank the top 50 down to the top 5

Bi-encoder embeddings are fast but coarse. A cross-encoder [[glossary/reranker|reranker]] is slow but accurate. Use them in series.

1. Retrieve top 50 by vector similarity.
2. Score each with a cross-encoder reranker (Cohere Rerank 3.5, BGE reranker, Voyage rerank-2.5).
3. Keep top 5 for the prompt.

The reranker sees the query and chunk together, which catches subtle relevance the embedding missed. Cost stays bounded because you only rerank 50, not the whole corpus.

## Evaluate retrieval and answer correctness separately

A golden set is two columns: question and the chunk IDs that contain the answer. A second sheet has question and the correct final answer.

- Retrieval metric: recall@k. Did the top k chunks include at least one chunk from the gold set? Track `recall@5`, `recall@10`, `recall@50`.
- Answer metric: correctness against the gold answer, scored by a judge model or by exact-match on extracted facts.

Track them separately. If recall@5 is 0.9 but answer correctness is 0.6, the problem is in generation. If recall@5 is 0.4, no prompt engineering will save you.

## Cite the source in the answer

Pass chunk IDs into the prompt; require the model to cite them inline.

```text
Context:
[1] (source: postgres-17-release-notes.md) ...
[2] (source: backend/postgres.md) ...

Question: How do I enable JIT in Postgres 17?

Answer with inline citations like [1] or [2] for each claim.
```

Citations are auditable. A user can click through; a developer can replay the retrieval and see whether the model used the chunk it cited.

## Filter by freshness when the topic moves

Stale chunks poison the answer. Attach `last_updated` to every chunk and filter at query time when the question is time-sensitive ("latest", "current", "in 2026").

- Hard filter: drop chunks older than N months for time-sensitive queries.
- Soft filter: rerank with a freshness boost, so older chunks are not excluded but lose ties.

The classifier is upstream: detect time sensitivity from the query, then pick the filter mode.

## Cache embeddings and retrievals

Re-embedding the same chunk is waste; re-retrieving the same query is waste.

- Cache embeddings by `(model_id, model_version, sha256(text))`. See [[ai-agents/embeddings]] for cache-key rules.
- Cache retrievals by `(query_text_normalized, filters)` with a short TTL. See [[glossary/semantic-cache]] for the lookup-by-meaning variant.

A warm cache cuts RAG latency in half and embedding spend by an order of magnitude.

## Related

- [[ai-agents/embeddings]]
- [[backend/chromadb]]
- [[prompt-engineering/prompt-design]]
- [[ai-agents/mcp-servers]]