Answers with sources: a stand-in model and citations

An answer a customer can trust says where it came from. The response keeps the chunks it used, and a stand-in model shows the flow without a key.

Exampleextractive_llm.py, part 1

import re

from llama_index.core.llms import CompletionResponse, CustomLLM, LLMMetadata
from llama_index.core.llms.callbacks import llm_completion_callback


def stems(text):
    """Words longer than three letters, cut to five letters, so refund and refunds match."""
    return {w[:5] for w in re.findall(r"[a-z0-9-]+", text.lower()) if len(w) > 3}


class ExtractiveLLM(CustomLLM):
    """Answers with the context sentence that shares most words with the question."""

    @property
    def metadata(self):
        return LLMMetadata(model_name="extractive")

Exampleextractive_llm.py, part 2

    @llm_completion_callback()
    def complete(self, prompt, formatted=False, **kwargs):
        context = prompt.split("---------------------")[1]
        question = prompt.split("Query:")[1].split("Answer:")[0]
        asked = stems(question)
        sentences = [s.strip() for s in re.split(r"(?<=[.!?])\s+|\n+", context)]
        sentences = [s for s in sentences if s and not s.startswith("#") and ": " not in s[:20]]
        best = max(sentences, key=lambda s: len(asked & stems(s)), default="")
        if len(asked & stems(best)) < 2:
            return CompletionResponse(text="I could not find that in the documents.")
        return CompletionResponse(text=best)

    @llm_completion_callback()
    def stream_complete(self, prompt, formatted=False, **kwargs):
        yield self.complete(prompt)

A LlamaIndex model is a CustomLLM subclass with metadata and complete, as the custom LLM guide shows. This one reads the context and question out of the prompt from lesson 9, skips headings and metadata lines, and answers with the sentence sharing most words with the question. Words are compared by their first five letters, a crude way to make refund match refunds, and fewer than two shared words means it says it could not find an answer. A real model writes a better sentence; the flow around it is identical.

Example

from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")

import os

from llama_index.core import SimpleDirectoryReader


def only_name(path):
    return {"file_name": os.path.basename(path)}


documents = SimpleDirectoryReader("help", file_metadata=only_name).load_data()

from llama_index.core.node_parser import SentenceSplitter

nodes = SentenceSplitter(chunk_size=80, chunk_overlap=0).get_nodes_from_documents(documents)

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex(nodes)
from extractive_llm import ExtractiveLLM

query_engine = index.as_query_engine(llm=ExtractiveLLM(), similarity_top_k=2)
for question in ["How long until my refund money reaches my card?", "Is the LMP-204 lamp safe?"]:
    response = query_engine.query(question)
    sources = sorted({node.metadata["file_name"] for node in response.source_nodes})
    print(f"{response} [{', '.join(sources)}]")

response.source_nodes holds the retrieved chunks the answer was built from, with their metadata. Listing their file names after the answer is a citation a customer, or a support agent checking the answer, can follow.

A citation says what was retrieved, not what was used. Two chunks were retrieved for each question, from two different files for the refund question, and one sentence answered it. Showing every source is honest about what the model saw; showing only the one it quoted needs the model to say which one.

Try it yourself

Ask "What are your opening hours?".
Print each source node's score next to its file name.
Set Settings.llm = ExtractiveLLM() once instead of passing llm=.

PreviousQuery engines: what the model is actually sent

Every expert started right here.