Better retrieval (hybrid search, reranking, query rewriting)
Better retrieval is a set of techniques that change which chunks a RAG pipeline hands to the model: keyword search next to vector search, a rewritten question, a second scoring pass, and filters on metadata.
Last updated: 08 Oct, 2026 · LangChain 1.4
Vector search matches meaning, and it can miss an exact name. Keyword search matches exact words, and it misses a paraphrase. A question may also be worded in a way that matches nothing in the document. Each technique below fixes one of these, and each is shown on the same four-page, two-paper corpus as Building a RAG pipeline.
Searching by keyword with BM25
BM25 is a keyword ranking function: it scores a chunk higher when it contains the query's words, especially rare words, and when those words are not buried in a very long chunk. It needs no embeddings. BM25Retriever in langchain-community wraps it (pip install rank_bm25).
Creating a BM25 retriever
By default the retriever splits text on spaces, so BM25[ followed by a citation number is one token and never matches the query BM25. A preprocess_func that keeps only word characters fixes it.
import re
from langchain_community.retrievers import BM25Retriever
keyword = BM25Retriever.from_documents(
chunks, k=3, preprocess_func=lambda text: re.findall(r"\w+", text.lower())
)
docs = keyword.invoke("RepLLaMA")Combining both lists with reciprocal rank fusion
Scores from two retrievers are on different scales, so adding them makes no sense. Reciprocal rank fusion (RRF) ignores the scores and uses only the positions: a chunk gets 1 / (k + rank) from each list it appears in, the numbers are added, and chunks are sorted by the total. k is a damping constant, 60 in the original paper. A chunk that both retrievers rank high wins.
Fusing two ranked lists
def fuse(*result_lists, k=60):
scores, docs = {}, {}
for results in result_lists:
for rank, doc in enumerate(results, start=1):
scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
docs[doc.id] = doc
return [docs[i] for i in sorted(scores, key=scores.get, reverse=True)]LangChain also ships this as EnsembleRetriever (in the langchain-classic package); writing the few lines yourself shows what it does. The example scores six questions by the rank, within each retriever's top three, of the first chunk that contains the answer text (a dash means it was not in the top three). The right column is the fused list, cut to three.
import re
from langchain_community.retrievers import BM25Retriever
vector = store.as_retriever(search_kwargs={"k": 3})
keyword = BM25Retriever.from_documents(
chunks, k=3, preprocess_func=lambda text: re.findall(r"\w+", text.lower())
)
def fuse(*result_lists, k=60):
scores, docs = {}, {}
for results in result_lists:
for rank, doc in enumerate(results, start=1):
scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
docs[doc.id] = doc
return [docs[i] for i in sorted(scores, key=scores.get, reverse=True)]
def rank(docs, needle):
for position, doc in enumerate(docs, start=1):
if needle in doc.page_content:
return str(position)
return "-"
tests = [
("RepLLaMA", "RepLLaMA"),
("BM25", "BM25"),
("What is the inner-layer size of the feed-forward network?", "2048"),
("What stops the decoder from peeking at future words?", "illegal connections"),
("How is the training set made more varied with the help of another language model?", "data synthesis"),
("Which benchmarks does the model rank first on?", "CMTEB"),
]
print(f"{'question':44} vector keyword fused")
for question, needle in tests:
v, k = vector.invoke(question), keyword.invoke(question)
print(f"{question[:42]:44} {rank(v, needle):^6} {rank(k, needle):^7} {rank(fuse(v, k)[:3], needle):^5}")
only = store.similarity_search("What is the model built on?", k=3, filter=lambda d: d.metadata["source"] == "qzhou.pdf")
print("filtered to qzhou.pdf:", [(d.id, d.metadata["source"]) for d in only])question vector keyword fused
RepLLaMA - 1 3
BM25 - 1 3
What is the inner-layer size of the feed-f 1 2 1
What stops the decoder from peeking at fut 2 3 2
How is the training set made more varied w - 3 -
Which benchmarks does the model rank first 1 1 1
filtered to qzhou.pdf: [('c7', 'qzhou.pdf'), ('c9', 'qzhou.pdf'), ('c13', 'qzhou.pdf')]Reading the rank table
- Vector search had the right chunk in its top three for 3 of the 6 questions. It missed the two exact names,
RepLLaMAandBM25, and the training-set question. - Keyword search had it for all 6, first for both names, but only second for the feed-forward question and third for the decoder question, where the question shares few words with the answer.
- The fused list had it for 5 of the 6: both names at rank 3, the feed-forward and benchmark questions at rank 1, the decoder question at rank 2. It lost the training-set question that keyword search had at rank 3: that chunk appears in one list only, at rank 3, so its score is 1/63, and the three places in the fused top three went to chunks with a score at least as high.
- Fusion does not always win. On this small corpus keyword search alone found the most; fusion beat vector search (5 against 3) but not keyword search. Its value is that one retriever's blind spot does not decide the result.
- The last line is the metadata filter: three chunks, all from
qzhou.pdf.
Filtering by metadata
The last two lines of the example search only the chunks whose metadata says qzhou.pdf. The metadata from Loading and chunking documents is what makes this possible: filter takes a function that sees each chunk's metadata and decides whether it may be returned. Real stores accept the same idea as a dictionary or a query, depending on the store.
Rewriting the question and reranking the results
Two more steps use the model. Query rewriting asks the model for several differently worded searches of the same question; the lists are fused. Reranking takes a longer candidate list and has the model score each chunk against the question, so the best few come first. Production systems often use a dedicated cross-encoder reranker for this step; the example uses the chat model as the scorer, with structured output so that the scores come back as a list of integers. Groq's openai/gpt-oss-120b needs method="json_schema" for that.
Rewriting with structured output
from pydantic import BaseModel
class Rewrites(BaseModel):
queries: list[str]
rewriter = model.with_structured_output(Rewrites, method="json_schema")
queries = rewriter.invoke("Rewrite this question as 3 short search queries: " + question).queriesScoring each candidate
class Scores(BaseModel):
scores: list[int]
grader = model.with_structured_output(Scores, method="json_schema")
scores = grader.invoke("Score each passage from 0 to 10 for how well it answers the question...").scoresfrom pydantic import BaseModel
from langchain.chat_models import init_chat_model
model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
retriever = store.as_retriever(search_kwargs={"k": 3})
class Rewrites(BaseModel):
queries: list[str]
class Scores(BaseModel):
scores: list[int]
question = "What did text embeddings evolve from?"
rewrites = model.with_structured_output(Rewrites, method="json_schema").invoke(
"Rewrite this question as 3 short search queries, each worded differently, "
"as they might appear in a research paper.\nQuestion: " + question
).queries
print(rewrites)
def fuse(*result_lists, k=60):
scores, docs = {}, {}
for results in result_lists:
for rank, doc in enumerate(results, start=1):
scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
docs[doc.id] = doc
return [docs[i] for i in sorted(scores, key=scores.get, reverse=True)]
candidates = fuse(*[retriever.invoke(q) for q in [question] + rewrites])[:5]
print("fused:", [d.id for d in candidates])
passages = "\n\n".join(f"Passage {i}: {d.page_content}" for i, d in enumerate(candidates))
graded = model.with_structured_output(Scores, method="json_schema").invoke(
f"Score each passage from 0 to 10 for how well it answers the question.\nQuestion: {question}\n\n{passages}"
).scores
reranked = [d for _, d in sorted(zip(graded, candidates), key=lambda pair: -pair[0])]
print("scores:", graded)
print("reranked:", [d.id for d in reranked])
print("top chunk:", repr(reranked[0].page_content[:70]))['origin of text embeddings', 'historical development of text embeddings', 'predecessors of text embeddings'] fused: ['c10', 'c11', 'c12'] scores: [9, 2, 1] reranked: ['c10', 'c11', 'c12'] top chunk: 'QZhou-Embedding Technical Report\n Kingsoft AI\nin terms of real-time pe'
Reading the rewrite and the rerank
- The model rewrote the question three ways:
origin of text embeddings,historical development of text embeddingsandpredecessors of text embeddings. - All four searches, the original and the three rewrites, returned the same chunks, so the fused list has three entries: c10, c11 and c12.
- The grader scored them 9, 2 and 1. Chunk c10 is the one that says text embeddings evolved from early sparse representations such as BM25, and the grader separated it clearly from the other two.
- The order did not change, because the fused list already had c10 first. Reranking pays when the right chunk sits low in a longer candidate list; here it confirms the order and shows the margin.
The techniques side by side
| Technique | Fixes | Costs | Skip it when |
|---|---|---|---|
| Hybrid search (BM25 + vectors) | Exact names and codes that embeddings blur | A second index to build and keep in step | Questions are always paraphrases |
| Metadata filter | Answers drawn from the wrong file | Metadata must be clean when indexing | There is one document |
| Query rewriting | A question worded unlike the text | One extra model call per question | Users already ask precise questions |
| Reranking | The right chunk found but ranked too low | One model call over the candidates, or a reranker service | The first k are already right |
Maximal marginal relevance (search_type="mmr") | Near-duplicate chunks filling the top k | A larger fetch before it selects | Overlap is small |
Where you use better retrieval
- Technical documents. Part numbers, function names and error codes are exact tokens; keyword search finds them where embeddings may not.
- Mixed archives. A metadata filter on department, year or file keeps a question about one policy away from another.
- Chat questions. Users type short, vague questions; a rewrite turns them into the words a document would use.
Related
- Previous: Building a RAG pipeline
- Next: Conversational RAG
- Go deeper: Retrieval tool
- Change
k=3tok=5in both retrievers and read whether the dash in the table disappears. - Add a seventh test question that uses a rare word from the QZhou page, such as
Jasper, and see which retriever finds it first. - Change the grader prompt to ask for a score from 0 to 3 and read how the ordering changes.
Every expert started right here.