The citation a chunk can build

A retrieval answer is only trustworthy if the reader can check it. Every chunk already carries where it came from: the file, the headings above it, and, for a PDF, the page. This lesson turns that into one line of citation.

Three pieces of provenance

document = DocumentConverter().convert("refunds.html").document
chunk = list(HybridChunker(max_tokens=64).chunk(document))[1]
print(chunk.meta.origin.filename)
print(chunk.meta.headings)

Two of the three come straight off the chunk's metadata: origin.filename from lesson 23, and headings from lesson 24. Together they already place this chunk under Refunds > How long it takes in refunds.html.

The page, when there is one

document = DocumentConverter().convert("returns.pdf").document
chunk = [c for c in HybridChunker(max_tokens=64).chunk(document)
         if "damaged" in c.text.lower()][0]
pages = sorted({prov.page_no for item in chunk.meta.doc_items for prov in item.prov})
print(pages)

The page is not on the chunk directly. A chunk was built from one or more document items, listed in chunk.meta.doc_items, and each item carries the provenance from lesson 15, with its page_no. Collecting the page of every item, and dropping duplicates with a set, gives the pages the chunk spans: page 2 for the damaged-items answer.

An HTML or Markdown chunk has no pages, so prov is empty and the set comes out empty too. That is why the page is gathered rather than read from a fixed place: the same code gives a page for the PDF and nothing for the others, instead of failing on the file that has no pages.

where()

def where(chunk):
    trail = " > ".join(chunk.meta.headings or ["(no heading)"])
    pages = sorted({prov.page_no for item in chunk.meta.doc_items for prov in item.prov})
    seen = "".join(f", page {n}" for n in pages)
    return f"{chunk.meta.origin.filename} > {trail}{seen}"

def where(chunk):
    trail = " > ".join(chunk.meta.headings or ["(no heading)"])
    pages = sorted({prov.page_no for item in chunk.meta.doc_items for prov in item.prov})
    seen = "".join(f", page {n}" for n in pages)
    return f"{chunk.meta.origin.filename} > {trail}{seen}"


for name in ["refunds.html", "returns.pdf"]:
    document = DocumentConverter().convert(name).document
    for chunk in HybridChunker(max_tokens=64).chunk(document):
        print(where(chunk))
        break

where joins the three into one string: the filename, the heading trail, and a page only when there is one. This is the citation the handbook prints under every answer, and it is assembled entirely from metadata the chunk already carried.