---
title: "RAG: Reranking"
slug: "rag-reranking"
category: "ai-agents"
tags: ["ai-agents", "rag", "reranking", "cross-encoder"]
status: "stable"
last_updated: 2026-05-29
summary: "Cross-encoder rerankers trim the top-50 candidate pool to a top-5 prompt. Cohere Rerank 3.5, BGE-Reranker, Voyage rerank-2.5, eval-driven selection."
related: ["[[rag]]", "[[rag-retrieval]]", "[[rag-chunking]]", "[[rag-eval]]", "[[embeddings]]"]
---

## Overview

Rerankers fix the precision that bi-encoder retrieval cannot. Embedding models compare query and chunk independently, then take a dot product; cross-encoders read query and chunk together and score the pair. The pair-aware score is more accurate and slower, which is exactly why reranking sits between retrieval and the prompt. For the retrieval step that feeds the reranker, see [[ai-agents/rag-retrieval]].

## Retrieve broad, rerank narrow

The pattern is two stages: a cheap recall stage that returns 30 to 100 candidates, then an accurate precision stage that returns the 3 to 8 chunks the prompt actually sees.

```text
query
  → dense retrieval top 30
  → sparse retrieval top 30
  → RRF merge to 50
  → cross-encoder rerank to top 5
  → prompt
```

Skipping the rerank step is the most common reason a "well-tuned" RAG system answers wrong on borderline queries. The embedding got close; the reranker would have closed the gap.

## Pick a hosted reranker for the easy path

Three production-ready hosted rerankers in 2026.

- Cohere Rerank 3.5 (`rerank-v3.5`): strong English baseline, multilingual, 4096-token context, low latency, simple billing.
- Voyage rerank-2.5 (and `rerank-2.5-lite`): pairs well with Voyage 4 embeddings; best when the embedding side is already Voyage. See [[ai-agents/embeddings]].
- Jina Reranker v2: open-weight option with a hosted API; good price per million tokens.

Run a bake-off on the eval suite before committing. The right reranker is the one that wins on your golden set, not the one with the best benchmark in a paper. See [[ai-agents/rag-eval]].

## Self-host BGE-Reranker when latency or data residency demands it

BGE-Reranker (`bge-reranker-large`, `bge-reranker-v2-m3`) is the open-weight default. Run it on a single GPU; throughput hits hundreds of pairs per second on an A10 or L4.

```python
from FlagEmbedding import FlagReranker

reranker = FlagReranker("BAAI/bge-reranker-large", use_fp16=True)
scores = reranker.compute_score([[query, chunk] for chunk in candidates])
ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
```

- Pin the model version in your image; reranker weights change.
- Batch pairs aggressively; GPU latency drops 10x against unbatched calls.

Self-host when the corpus is sensitive, the latency budget is tight, or the volume makes a hosted API uneconomic.

## Combine retrievers with reciprocal rank fusion before rerank

When the candidate pool comes from multiple retrievers (dense, BM25, knowledge graph), merge with RRF before reranking. The reranker is expensive; do not score the same chunk twice.

```python
def rrf(rankings, k=60):
    scores = {}
    for r in rankings:
        for rank, doc_id in enumerate(r):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)[:50]
```

RRF is parameter-light (`k=60` default), works with any number of retrievers, and does not need score calibration.

## Budget the reranker on a latency target

Reranking is the slowest hop in the pipeline that is still cheap enough to keep. Set a budget; respect it.

- Hosted Rerank API: 50 to 200 ms for 50 pairs. Plan for the long tail at 500 ms.
- Self-hosted BGE on a GPU: 20 to 100 ms for 50 pairs at FP16, batched.
- CPU reranking: usable for prototypes only.

If the budget is under 150 ms total, drop to top-25 candidates or use a smaller reranker. Do not skip reranking; cut its scope.

## Eval-drive the reranker choice

Reranker quality varies by corpus far more than leaderboards suggest. Pick with numbers, not vendor pitch.

- Score `recall@5` and `nDCG@5` against the gold set on three rerankers.
- Score the same chunks with no reranker as the baseline.
- Pick the reranker with the largest lift at acceptable latency.

A reranker that adds 1 point of accuracy at 4x the cost loses. See [[ai-agents/rag-eval]] for the harness.

## Pass enough context, not too much

The reranked top-k goes into the prompt. More context is not always better.

- Start at top 5 chunks. Most generation models lose accuracy past 8 to 10 chunks per query.
- Order chunks by reranker score, highest first. Long-context models still weight the head of the prompt.
- Trim chunks to the relevant section when individual chunks are long; do not paste the full 800 tokens.

The reranker did the hard work. The prompt should reflect that. See [[ai-agents/rag-citations]] for how to format the chunks for generation.

## Related

- [[ai-agents/rag]]
- [[ai-agents/rag-retrieval]]
- [[ai-agents/rag-chunking]]
- [[ai-agents/rag-eval]]
- [[ai-agents/rag-citations]]
- [[ai-agents/embeddings]]