Reranking for RAG

Reranking is a retrieval optimization technique that reorders search results according to how relevant they are to a user's query. In retrieval-augmented generation (RAG), reranking is commonly placed after the initial retrieval stage and before the final context is sent to the language model.

A typical RAG system may retrieve dozens of candidate document chunks using keyword search, semantic search, or hybrid search. These methods are fast and designed to find a broad set of potentially relevant results. A reranker then examines the query and retrieved candidates in greater detail and produces a more precise ordering.

Reranking can therefore improve the quality of the context given to an LLM without requiring the entire retrieval system to use an expensive model for every document in the database.

What Is Reranking?

Reranking is the process of taking an existing list of search results and producing a new ranking based on a more detailed relevance calculation. The first retrieval system prioritizes speed and recall, while the reranker prioritizes precision among the retrieved candidates.

For example, a vector search system might retrieve 50 document chunks for a question. The first few results may be broadly related to the topic but not actually answer the question. A reranker can compare the query with each candidate and move the most useful chunks toward the top.

Reranking does not normally search the entire document collection again. It works on a candidate set that has already been retrieved.

Why Does RAG Need Reranking?

Retrieval is one of the most important parts of a RAG system. If the correct information is not included in the retrieved context, the language model cannot reliably use it to answer the question.

Initial retrieval systems are usually optimized for speed and recall. Vector search can identify documents with similar meaning, while keyword search can find exact terms. However, the highest-scoring retrieved document is not necessarily the document that best answers the specific question.

A reranker adds another relevance judgment between retrieval and generation. Instead of asking only whether a document is related to the query, the reranker attempts to determine how strongly the document matches the query.

The RAG Retrieval Pipeline

A basic RAG system often follows this sequence:

• User submits a question.
• The application processes the query.
• The query is searched against a document index.
• The retrieval system returns candidate documents.
• A reranker scores and reorders the candidates.
• The highest-quality candidates are selected.
• Selected content is inserted into the LLM context.
• The language model generates the final answer.

The reranker therefore sits between retrieval and context construction. Its job is not to generate the answer. Its job is to improve the ordering of information before generation.

Retrieval vs Reranking

Retrieval and reranking solve related but different problems. Retrieval answers the question: which documents could be relevant? Reranking answers a more precise question: which of these retrieved documents are most relevant to this particular query?

This distinction makes it possible to use a fast retrieval mechanism for a large collection and a slower, more accurate model for a much smaller candidate set.

For example, searching one million document chunks with a sophisticated reranking model would be expensive. Searching the million chunks with an efficient vector or lexical index and then reranking 50 or 100 candidates is much more practical.

How Rerankers Work

A reranker receives a query and one or more candidate documents. It evaluates the relationship between them and assigns a relevance score to each candidate. The candidates are then sorted according to those scores.

Query + Document 1 → relevance score
Query + Document 2 → relevance score
Query + Document 3 → relevance score
                  ↓
          Sort by relevance
                  ↓
            Top candidates

The model used for reranking can examine interactions between the query and document rather than relying only on independently generated embeddings. This can make it better at detecting whether a specific passage actually addresses the question.

Cross-Encoder Rerankers

A common reranking architecture is a cross-encoder. Instead of generating separate embeddings for the query and document and comparing them, a cross-encoder processes the query and candidate document together.

Because the model can directly examine the relationship between the query and document tokens, it can capture detailed interactions that a simple vector similarity calculation may miss.

This usually improves ranking quality, but it also makes cross-encoders more computationally expensive than embedding-based retrieval. Every query-document pair must be evaluated.

Bi-Encoders and Embedding Retrieval

Embedding models are often called bi-encoder-style retrieval systems because the query and document can be encoded independently. The resulting vectors can then be compared efficiently using a vector index.

This architecture is extremely useful for large-scale retrieval because document embeddings can be generated ahead of time and indexed. A new query only requires generating one query embedding before searching the index.

The trade-off is that independent embeddings provide less direct interaction between the query and each document than a cross-encoder can provide.

Cross-Encoder vs Bi-Encoder

This is why a common architecture uses an embedding model for initial retrieval and a cross-encoder-style model for reranking.

How Many Documents Should Be Reranked?

The number of candidates sent to a reranker is an important tuning parameter. If the candidate set is too small, the correct document may never reach the reranking stage. If it is too large, latency and compute costs increase.

A common strategy is to retrieve more documents than will ultimately be placed into the LLM context. For example, an application might retrieve 50 candidates, rerank them, and then select the best 5 to 10 chunks.

The correct numbers depend on the dataset, query complexity, document size, reranker performance, latency requirements, and context window of the target model.

Reranking and Top-K Retrieval

Top-K is the number of results returned by a retrieval or ranking stage. In a RAG pipeline, you may have several different K values.

• Initial retrieval K: number of candidates retrieved from the search index.
• Reranking K: number of candidates evaluated by the reranker.
• Final context K: number of documents or chunks supplied to the LLM.

For example, a pipeline could retrieve 50 candidates, rerank all 50, and pass the top 8 chunks to the language model. These numbers are illustrative rather than universal defaults.

Reranking with Hybrid Search

Reranking is especially effective when the candidate set comes from hybrid search. Keyword search can retrieve documents containing exact terms, while vector search can retrieve semantically related documents.

The two result lists can be combined using score-based fusion or rank-based methods such as Reciprocal Rank Fusion. The resulting candidate set can then be passed to a reranker.

Query
  ├── Keyword Search
  ├── Vector Search
  ↓
Result Fusion
  ↓
Reranker
  ↓
Top Context
  ↓
LLM

This layered approach combines broad retrieval with detailed relevance scoring and is often stronger than relying on a single retrieval signal.

Reranking for Semantic Search

Reranking can also improve a semantic-search-only system. Vector similarity provides a fast way to identify documents with related meanings, but semantic similarity does not always mean that a document directly answers the user's question.

For example, a query about configuring authentication might retrieve several documents about authentication because they are semantically close. A reranker can help distinguish a document specifically describing configuration from documents that only discuss authentication conceptually.

Reranking in Long Documents

Long documents are often divided into chunks before being indexed. This creates a large number of candidates and makes ranking quality particularly important.

A reranker can evaluate individual chunks against the query and move the most useful passages to the top. However, the system should also preserve enough surrounding context when a relevant chunk depends on information from nearby sections.

In some applications, it is useful to store document and section metadata with each chunk so that the final context can include the title or surrounding heading along with the retrieved passage.

Reranking and Context Windows

Language models have finite context windows. Even when a model supports a very large context, sending large amounts of irrelevant information can increase cost and potentially make it harder for the model to focus on the most useful evidence.

Reranking helps reduce the candidate set before generation. Instead of sending dozens of loosely related chunks, the application can select a smaller group of high-quality candidates.

This makes reranking useful not only for retrieval accuracy but also for context management and inference efficiency.

Reranking and Hallucinations

Better retrieval can reduce some causes of hallucination because the LLM receives more relevant supporting information. However, reranking does not eliminate hallucinations by itself.

A reranker only improves the ordering of retrieved information. It cannot create missing information, verify every factual statement, or guarantee that the language model will faithfully use the selected context.

For high-reliability applications, reranking should therefore be combined with good document preparation, retrieval evaluation, source attribution, prompt design, and output validation where appropriate.

Reranking Scores

A reranker usually produces a relevance score for each query-document pair. These scores can be used to sort candidates from most relevant to least relevant.

The exact numerical meaning of a score depends on the model and implementation. A score from one reranker should not automatically be compared with a score from another reranker.

Using a Relevance Threshold

Some systems use a minimum relevance threshold after reranking. Candidates below the threshold can be removed instead of being sent to the LLM.

Thresholds can be useful when the system needs to distinguish between relevant and irrelevant results, but they should be calibrated against representative queries. A threshold that works for one dataset may be inappropriate for another.

In some cases, a fixed top-K strategy is simpler and more stable. Another approach is to use both a maximum number of results and a relevance threshold so that obviously poor candidates are excluded without allowing the context to grow indefinitely.

Reranking and Metadata

Metadata can improve retrieval before reranking and can also influence which candidates are eligible for reranking. Useful metadata includes document type, category, language, version, publication date, product, user permissions, and tenant ID.

For example, a developer asking about a specific framework version should not receive documentation for a completely different version simply because the text is semantically similar.

Authorization filtering is even more important. A reranker should never be used as a substitute for access control. Documents that a user is not allowed to access should be excluded before they can enter the model context.

Reranking and Query Rewriting

Query rewriting and reranking solve different problems. Query rewriting changes or expands the query before retrieval, while reranking evaluates the retrieved candidates after retrieval.

They can be used together. A rewritten query may improve recall, while reranking can improve precision among the resulting candidates.

Reranking and Multiple Queries

Advanced RAG systems sometimes generate multiple search queries from a user's original question. Each query can retrieve its own candidate set, and the combined candidates can then be deduplicated and reranked.

This can improve recall for complex questions because different query formulations may expose different parts of the knowledge base. The additional retrieval and reranking work increases system complexity and latency, so it should be used when the improvement is measurable.

Reranking Latency

The primary trade-off of reranking is latency. Initial vector and keyword retrieval can be highly optimized using indexes, while a reranker may need to process every query-candidate pair individually.

Running the retrieval systems in parallel and limiting the candidate set can reduce the impact. Reranking can also be performed only when the query is difficult or when initial retrieval confidence is low.

For interactive applications, benchmark the complete pipeline rather than measuring only the reranker. The goal is to improve answer quality without making the user experience unnecessarily slow.

Reranking Cost Optimization

When a reranker is provided as an external API, cost may depend on the amount of text or number of documents processed. Sending unnecessarily large chunks or too many candidates can therefore increase expenses.

• Retrieve a reasonable candidate set instead of reranking the entire corpus.
• Avoid unnecessarily large document chunks.
• Use metadata filters before reranking.
• Deduplicate candidates before sending them to the reranker.
• Measure whether reranking improves answer quality enough to justify its cost.
• Cache appropriate repeated retrieval operations when possible.
• Use cheaper retrieval methods for broad candidate generation.

How to Evaluate a Reranker

A reranker should be evaluated using a dataset containing representative queries and known relevant documents. Compare the ranking before and after reranking instead of assuming that a more sophisticated model will automatically improve results.

Useful ranking metrics include Precision@K, Recall@K, Mean Reciprocal Rank, and Normalized Discounted Cumulative Gain. The appropriate metric depends on the retrieval task and whether there can be multiple relevant documents for each query.

For RAG, it is also useful to evaluate downstream answer quality. A reranker may improve retrieval metrics but have little effect on the final answers if the original retrieval was already sufficient or if another component is the primary bottleneck.

A/B Testing Reranking

For a production application, A/B testing can provide stronger evidence than a small collection of manual examples. One group can use the existing retrieval pipeline while another uses the same pipeline with reranking.

Measure retrieval quality, answer quality, latency, token usage, error rates, and infrastructure cost. This makes it possible to determine whether reranking provides enough value for the application.

Common Reranking Mistakes

• Reranking too few candidates.
• Reranking far too many candidates.
• Using very large chunks that contain multiple unrelated topics.
• Ignoring metadata and authorization filters.
• Treating reranking scores as universal probabilities.
• Assuming reranking can recover documents that initial retrieval never found.
• Evaluating only a few manually selected queries.
• Ignoring latency and API costs.
• Sending all reranked results to the LLM instead of selecting a focused context.
• Using a reranker without measuring whether it improves the final application.

An Important Limitation of Reranking

A reranker cannot rank a document that was never retrieved. This is one of the most important limitations to understand.

If the correct document is missing from the initial candidate set, a perfect reranker cannot select it. Improving initial retrieval recall is therefore still necessary.

For this reason, RAG optimization should usually consider the complete retrieval pipeline: document quality, chunking, embeddings, keyword search, semantic search, hybrid retrieval, candidate count, reranking, and final context selection.

A Practical RAG Reranking Strategy

A practical starting architecture is to use hybrid retrieval or semantic retrieval to produce a moderately sized candidate set, rerank those candidates, and then pass only the strongest chunks to the LLM.

1. Query
2. Keyword + semantic retrieval
3. Merge and deduplicate candidates
4. Apply metadata and access filters
5. Retrieve a moderate candidate set
6. Rerank candidates
7. Select top relevant chunks
8. Build LLM context
9. Generate answer
10. Evaluate and monitor

This architecture is flexible enough to support different retrieval technologies and reranking models. The exact values for candidate counts, thresholds, and context size should be determined through evaluation rather than copied as universal defaults.

When Should You Add Reranking?

Reranking is most useful when initial retrieval already finds the correct information but frequently places less relevant documents above it. If the correct documents are not being retrieved at all, improving indexing, chunking, embeddings, query processing, or recall should come first.

• Add reranking when top retrieval results contain many near-matches.
• Add reranking when semantic similarity is not precise enough.
• Add reranking when hybrid retrieval produces strong but differently ranked candidates.
• Add reranking when context limits require selecting a small number of high-quality chunks.
• Do not rely on reranking to solve poor indexing or missing documents.

Reranking Best Practices

• Use fast retrieval for high recall.
• Use reranking for high precision.
• Keep the reranking candidate set manageable.
• Combine lexical and semantic retrieval when exact and conceptual matches both matter.
• Filter documents by authorization before they reach the LLM.
• Use metadata to improve candidate selection.
• Choose chunk sizes appropriate for your documents and queries.
• Evaluate reranking using representative search queries.
• Measure both retrieval quality and final answer quality.
• Monitor latency and cost in production.
• Tune candidate counts and thresholds using real data.
• Remember that reranking cannot recover missing candidates.

Frequently Asked Questions

Conclusion

Reranking is a powerful second-stage retrieval technique for RAG systems. Instead of asking an expensive relevance model to search an entire knowledge base, a fast retrieval system first produces a manageable candidate set and the reranker then focuses on ordering those candidates more precisely.

Cross-encoder-style rerankers are particularly useful because they can examine the query and document together. Combined with semantic search, keyword search, or hybrid retrieval, reranking can improve the quality of the information ultimately provided to the LLM.

The key is to treat reranking as one component of a larger retrieval pipeline. Good document preparation, chunking, embeddings, metadata filtering, high-recall retrieval, reranking, context selection, and evaluation all contribute to RAG quality. Reranking cannot compensate for documents that were never retrieved, but when initial retrieval already finds useful candidates and the problem is poor ordering, it can make a significant difference.