Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Better retrieval (hybrid search, reranking, query rewriting)

Better retrieval is a set of techniques that change which chunks a RAG pipeline hands to the model: keyword search next to vector search, a rewritten question, a second scoring pass, and filters on metadata.

Last updated: 08 Oct, 2026 · LangChain 1.4

Vector search matches meaning, and it can miss an exact name. Keyword search matches exact words, and it misses a paraphrase. A question may also be worded in a way that matches nothing in the document. Each technique below fixes one of these, and each is shown on the same four-page, two-paper corpus as Building a RAG pipeline.

A question may first be rewritten by an LLM. It is then searched two ways, by keyword with BM25 and by vector similarity. The two ranked lists are fused with reciprocal rank fusion, an LLM can rerank the fused list, and the top chunks go to the LLM that writes the answer.

Searching by keyword with BM25

BM25 is a keyword ranking function: it scores a chunk higher when it contains the query's words, especially rare words, and when those words are not buried in a very long chunk. It needs no embeddings. BM25Retriever in langchain-community wraps it (pip install rank_bm25).

Creating a BM25 retriever

By default the retriever splits text on spaces, so BM25[ followed by a citation number is one token and never matches the query BM25. A preprocess_func that keeps only word characters fixes it.

python
import re
from langchain_community.retrievers import BM25Retriever

keyword = BM25Retriever.from_documents(
    chunks, k=3, preprocess_func=lambda text: re.findall(r"\w+", text.lower())
)
docs = keyword.invoke("RepLLaMA")

Combining both lists with reciprocal rank fusion

Scores from two retrievers are on different scales, so adding them makes no sense. Reciprocal rank fusion (RRF) ignores the scores and uses only the positions: a chunk gets 1 / (k + rank) from each list it appears in, the numbers are added, and chunks are sorted by the total. k is a damping constant, 60 in the original paper. A chunk that both retrievers rank high wins.

Reciprocal rank fusion, with k = 60

Fusing two ranked lists

python
def fuse(*result_lists, k=60):
    scores, docs = {}, {}
    for results in result_lists:
        for rank, doc in enumerate(results, start=1):
            scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
            docs[doc.id] = doc
    return [docs[i] for i in sorted(scores, key=scores.get, reverse=True)]

LangChain also ships this as EnsembleRetriever (in the langchain-classic package); writing the few lines yourself shows what it does. The example scores six questions by the rank, within each retriever's top three, of the first chunk that contains the answer text (a dash means it was not in the top three). The right column is the fused list, cut to three.

ExampleAPI keyStarts from the four-page store of the previous lesson; run on rank-bm25 0.2.2 and langchain-community 0.4.2
import re
from langchain_community.retrievers import BM25Retriever

vector = store.as_retriever(search_kwargs={"k": 3})
keyword = BM25Retriever.from_documents(
    chunks, k=3, preprocess_func=lambda text: re.findall(r"\w+", text.lower())
)

def fuse(*result_lists, k=60):
    scores, docs = {}, {}
    for results in result_lists:
        for rank, doc in enumerate(results, start=1):
            scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
            docs[doc.id] = doc
    return [docs[i] for i in sorted(scores, key=scores.get, reverse=True)]

def rank(docs, needle):
    for position, doc in enumerate(docs, start=1):
        if needle in doc.page_content:
            return str(position)
    return "-"

tests = [
    ("RepLLaMA", "RepLLaMA"),
    ("BM25", "BM25"),
    ("What is the inner-layer size of the feed-forward network?", "2048"),
    ("What stops the decoder from peeking at future words?", "illegal connections"),
    ("How is the training set made more varied with the help of another language model?", "data synthesis"),
    ("Which benchmarks does the model rank first on?", "CMTEB"),
]
print(f"{'question':44} vector keyword fused")
for question, needle in tests:
    v, k = vector.invoke(question), keyword.invoke(question)
    print(f"{question[:42]:44} {rank(v, needle):^6} {rank(k, needle):^7} {rank(fuse(v, k)[:3], needle):^5}")

only = store.similarity_search("What is the model built on?", k=3, filter=lambda d: d.metadata["source"] == "qzhou.pdf")
print("filtered to qzhou.pdf:", [(d.id, d.metadata["source"]) for d in only])

Reading the rank table

  • Vector search had the right chunk in its top three for 3 of the 6 questions. It missed the two exact names, RepLLaMA and BM25, and the training-set question.
  • Keyword search had it for all 6, first for both names, but only second for the feed-forward question and third for the decoder question, where the question shares few words with the answer.
  • The fused list had it for 5 of the 6: both names at rank 3, the feed-forward and benchmark questions at rank 1, the decoder question at rank 2. It lost the training-set question that keyword search had at rank 3: that chunk appears in one list only, at rank 3, so its score is 1/63, and the three places in the fused top three went to chunks with a score at least as high.
  • Fusion does not always win. On this small corpus keyword search alone found the most; fusion beat vector search (5 against 3) but not keyword search. Its value is that one retriever's blind spot does not decide the result.
  • The last line is the metadata filter: three chunks, all from qzhou.pdf.

Filtering by metadata

The last two lines of the example search only the chunks whose metadata says qzhou.pdf. The metadata from Loading and chunking documents is what makes this possible: filter takes a function that sees each chunk's metadata and decides whether it may be returned. Real stores accept the same idea as a dictionary or a query, depending on the store.

Rewriting the question and reranking the results

Two more steps use the model. Query rewriting asks the model for several differently worded searches of the same question; the lists are fused. Reranking takes a longer candidate list and has the model score each chunk against the question, so the best few come first. Production systems often use a dedicated cross-encoder reranker for this step; the example uses the chat model as the scorer, with structured output so that the scores come back as a list of integers. Groq's openai/gpt-oss-120b needs method="json_schema" for that.

Rewriting with structured output

python
from pydantic import BaseModel

class Rewrites(BaseModel):
    queries: list[str]

rewriter = model.with_structured_output(Rewrites, method="json_schema")
queries = rewriter.invoke("Rewrite this question as 3 short search queries: " + question).queries

Scoring each candidate

python
class Scores(BaseModel):
    scores: list[int]

grader = model.with_structured_output(Scores, method="json_schema")
scores = grader.invoke("Score each passage from 0 to 10 for how well it answers the question...").scores
ExampleAPI keyStarts from the four-page store of the previous lesson; run on Groq openai/gpt-oss-120b
from pydantic import BaseModel
from langchain.chat_models import init_chat_model

model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
retriever = store.as_retriever(search_kwargs={"k": 3})

class Rewrites(BaseModel):
    queries: list[str]

class Scores(BaseModel):
    scores: list[int]

question = "What did text embeddings evolve from?"
rewrites = model.with_structured_output(Rewrites, method="json_schema").invoke(
    "Rewrite this question as 3 short search queries, each worded differently, "
    "as they might appear in a research paper.\nQuestion: " + question
).queries
print(rewrites)

def fuse(*result_lists, k=60):
    scores, docs = {}, {}
    for results in result_lists:
        for rank, doc in enumerate(results, start=1):
            scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
            docs[doc.id] = doc
    return [docs[i] for i in sorted(scores, key=scores.get, reverse=True)]

candidates = fuse(*[retriever.invoke(q) for q in [question] + rewrites])[:5]
print("fused:", [d.id for d in candidates])

passages = "\n\n".join(f"Passage {i}: {d.page_content}" for i, d in enumerate(candidates))
graded = model.with_structured_output(Scores, method="json_schema").invoke(
    f"Score each passage from 0 to 10 for how well it answers the question.\nQuestion: {question}\n\n{passages}"
).scores
reranked = [d for _, d in sorted(zip(graded, candidates), key=lambda pair: -pair[0])]
print("scores:", graded)
print("reranked:", [d.id for d in reranked])
print("top chunk:", repr(reranked[0].page_content[:70]))

Reading the rewrite and the rerank

  • The model rewrote the question three ways: origin of text embeddings, historical development of text embeddings and predecessors of text embeddings.
  • All four searches, the original and the three rewrites, returned the same chunks, so the fused list has three entries: c10, c11 and c12.
  • The grader scored them 9, 2 and 1. Chunk c10 is the one that says text embeddings evolved from early sparse representations such as BM25, and the grader separated it clearly from the other two.
  • The order did not change, because the fused list already had c10 first. Reranking pays when the right chunk sits low in a longer candidate list; here it confirms the order and shows the margin.

The techniques side by side

TechniqueFixesCostsSkip it when
Hybrid search (BM25 + vectors)Exact names and codes that embeddings blurA second index to build and keep in stepQuestions are always paraphrases
Metadata filterAnswers drawn from the wrong fileMetadata must be clean when indexingThere is one document
Query rewritingA question worded unlike the textOne extra model call per questionUsers already ask precise questions
RerankingThe right chunk found but ranked too lowOne model call over the candidates, or a reranker serviceThe first k are already right
Maximal marginal relevance (search_type="mmr")Near-duplicate chunks filling the top kA larger fetch before it selectsOverlap is small

Where you use better retrieval

  • Technical documents. Part numbers, function names and error codes are exact tokens; keyword search finds them where embeddings may not.
  • Mixed archives. A metadata filter on department, year or file keeps a question about one policy away from another.
  • Chat questions. Users type short, vague questions; a rewrite turns them into the words a document would use.
Watch out. Each technique adds a step that can also be wrong, and the effect depends on the corpus. On this 15-chunk corpus keyword search is unusually strong because the questions share words with the text. Measure retrieval on questions from your own documents, as the Evaluating RAG lesson does, before keeping a step.
Try it yourself
  • Change k=3 to k=5 in both retrievers and read whether the dash in the table disappears.
  • Add a seventh test question that uses a rare word from the QZhou page, such as Jasper, and see which retriever finds it first.
  • Change the grader prompt to ask for a score from 0 to 3 and read how the ordering changes.

Every expert started right here.