Refusing when nothing fits: similarity cutoffs

Top-k retrieval always returns something, even for a question no document answers. A similarity cutoff drops weak chunks so the assistant can refuse.

Example

from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")

import os

from llama_index.core import SimpleDirectoryReader


def only_name(path):
    return {"file_name": os.path.basename(path)}


documents = SimpleDirectoryReader("help", file_metadata=only_name).load_data()

from llama_index.core.node_parser import SentenceSplitter

nodes = SentenceSplitter(chunk_size=80, chunk_overlap=0).get_nodes_from_documents(documents)

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex(nodes)

retriever = index.as_retriever(similarity_top_k=2)
for question in ["How long does standard delivery take?", "What are your opening hours?"]:
    print(question, [(n.metadata["file_name"], round(n.score, 2)) for n in retriever.retrieve(question)])

No document mentions opening hours, yet two chunks came back for that question, just with lower scores. A model given those chunks might say it does not know, or might invent an answer from the nearest thing it was given.

A cutoff

Example

from llama_index.core.postprocessor import SimilarityPostprocessor
from extractive_llm import ExtractiveLLM

query_engine = index.as_query_engine(
    llm=ExtractiveLLM(),
    similarity_top_k=2,
    node_postprocessors=[SimilarityPostprocessor(similarity_cutoff=0.3)],
)
for question in ["How long does standard delivery take?", "What are your opening hours?"]:
    response = query_engine.query(question)
    print(question, "->", response, f"({len(response.source_nodes)} sources)")

A node postprocessor runs between retrieval and the model. SimilarityPostprocessor drops nodes scoring below the cutoff. For the opening hours question nothing was left, and LlamaIndex answered Empty Response without calling the model at all, which your application can turn into a clear refusal.

The cutoff is a number to measure, not guess. Scores depend on the embedding model and on how questions are phrased. Too high, and good answers are refused; too low, and nothing is. Lesson 16 shows how to test it on labelled questions.

Try it yourself

Try cutoffs of 0.1 and 0.5 on both questions.
Replace Empty Response with your own refusal message when response.source_nodes is empty.
Ask "Can I return a used bulb?" with the cutoff.

PreviousAnswers with sources: a stand-in model and citations

Little by little, you're building something great.