BM25: when exact words matter

Embeddings match meaning, and a part number has almost none. For codes, names and exact phrases, the keyword ranking BM25 finds what embeddings miss.

from llama_index.retrievers.bm25 import BM25Retriever

keyword = BM25Retriever.from_defaults(nodes=nodes, similarity_top_k=2)
semantic = index.as_retriever(similarity_top_k=2)

BM25 is a long-established keyword ranking: it scores a chunk higher when it contains the question's words, especially words that are rare across all chunks. It needs no embedding model. The BM25 integration builds its index from the same nodes as the vector index.

for question in ["LMP-204", "cable fault on my lamp", "Is reimbursement possible?"]:
    print(question)
    print("   keyword: ", [(n.metadata["file_name"], round(n.score, 2)) for n in keyword.retrieve(question)])
    print("   semantic:", [(n.metadata["file_name"], round(n.score, 2)) for n in semantic.retrieve(question)])

For LMP-204, keyword search scores the chunk naming that exact part highest, and the other lamp chunk lower because it only shares the LMP part of a different number. Semantic search ranks the same way with weaker scores: to an embedding model a part number is a string of characters with little meaning.

For "Is reimbursement possible?", the opposite: it shares no word with any chunk, so every keyword score is 0 and the order means nothing, while semantic search ranks both refund chunks first.

Each method fails where the other succeeds, which is the case for combining them in the next lesson. Catalogues, error codes, policy numbers and legal clause references are where customers' questions break pure semantic search.