Reranking: a second, closer look at the top results

An embedding compares question and chunk separately. A reranker reads them together and scores the fit more accurately, so it reorders only the top few.

from llama_index.core.postprocessor import SentenceTransformerRerank
from llama_index.core.schema import QueryBundle

reranker = SentenceTransformerRerank(model="cross-encoder/ms-marco-MiniLM-L-2-v2", top_n=2)

SentenceTransformerRerank uses a cross-encoder: a model that takes the question and a passage as one input and returns a relevance score. This small one downloads once, about 70 MB. top_n=2 keeps the best two after reordering.

question = "How do I send an item back?"
candidates = index.as_retriever(similarity_top_k=4).retrieve(question)
print("retrieved:", [n.get_content().replace("\n", " ")[:45] for n in candidates])

reranked = reranker.postprocess_nodes(candidates, query_bundle=QueryBundle(question))
print("reranked: ", [(n.get_content().replace("\n", " ")[:45], round(float(n.score), 2)) for n in reranked])

The retriever cast a wide net of four chunks and ranked a chunk about printing a label first and a delivery chunk second. The reranker read each against the question, moved the chunk that says to open the order and choose Return an item to the top, and dropped the delivery chunk from the best two. Its scores are on a different scale from similarity, negative numbers included: they are the cross-encoder's own relevance scores, useful for ordering.

The usual shape: retrieve generously, say 10 to 50 chunks, then rerank to the best 3 to 5 for the prompt. Retrieval is fast and approximate; reranking is slower and precise, and only ever sees a handful.