Curious builder0 XP earned · 300 to level 2

0 daysFinish a lesson to begin

Badge collection0 of 6 unlocked

31 small wins to finish your path

handbook.py, end to end

Every piece is written: the citation, the search, the report. Put them in one file and the shop's four documents in three formats become a handbook that answers a question and shows where the answer came from.

build()

python

def build(names=HANDBOOK):
    converter = DocumentConverter()
    chunker = HybridChunker(max_tokens=64)
    index, report = [], []
    for result in converter.convert_all([Path(n) for n in names], raises_on_error=False):
        name = result.input.file.name
        if result.status is not ConversionStatus.SUCCESS:
            report.append((name, result.status.name, "-"))
            continue
        for chunk in chunker.chunk(result.document):
            index.append(Entry(chunker.contextualize(chunk), chunk.text, where(chunk)))
        report.append((name, result.status.name, result.confidence.mean_grade.value))
    return index, report

build is lesson 29's loop with the chunker inside it. Each chunk becomes an Entry of three things: the contextualized text to search from lesson 28, the plain text to show, and the citation from lesson 27. It returns the index and the report together.

ask()

python

def ask(index, question):
    asked = words(question)
    best, position, entry = max(
        (len(asked & words(e.searchable)), -i, e) for i, e in enumerate(index))
    if best < 2:
        return "I could not find that in the handbook."
    return f"{entry.text}\n    [{entry.source}]"

ask scores every entry by word overlap and keeps the best, with -i breaking a tie in favour of the earlier chunk. Below two shared words it says it did not find the answer, rather than return the closest wrong one. Otherwise it prints the chunk text and, indented beneath, its citation.

Asking it

Example

from handbook import build, ask

index, report = build()
print(len(index), "chunks from", len(report), "files")
print(ask(index, "how long does a refund take?"))
print(ask(index, "can I return a damaged item?"))

Nine chunks from four files, and two answers each with a citation a reader can open: the refund table cites refunds.html under its headings, and the damaged-items answer cites page 2 of returns.pdf. The PDF answer proves the whole chain, because the page number came all the way from the provenance in lesson 15.

The whole file

python

import re
from collections import namedtuple
from pathlib import Path

from docling.chunking import HybridChunker
from docling.datamodel.base_models import ConversionStatus
from docling.document_converter import DocumentConverter

HANDBOOK = ["shipping.md", "refunds.html", "orders.csv", "returns.pdf"]
STOP = {"a", "an", "the", "is", "are", "do", "does", "how", "what", "when",
        "who", "i", "my", "of", "to", "for", "it", "and", "can", "you", "your"}
Entry = namedtuple("Entry", "searchable text source")


def where(chunk):
    trail = " > ".join(chunk.meta.headings or ["(no heading)"])
    pages = sorted({prov.page_no for item in chunk.meta.doc_items for prov in item.prov})
    seen = "".join(f", page {n}" for n in pages)
    return f"{chunk.meta.origin.filename} > {trail}{seen}"


def build(names=HANDBOOK):
    converter = DocumentConverter()
    chunker = HybridChunker(max_tokens=64)
    index, report = [], []
    for result in converter.convert_all([Path(n) for n in names], raises_on_error=False):
        name = result.input.file.name
        if result.status is not ConversionStatus.SUCCESS:
            report.append((name, result.status.name, "-"))
            continue
        for chunk in chunker.chunk(result.document):
            index.append(Entry(chunker.contextualize(chunk), chunk.text, where(chunk)))
        report.append((name, result.status.name, result.confidence.mean_grade.value))
    return index, report


def words(text):
    found = re.findall(r"[a-z0-9]+", text.lower())
    return {word.rstrip("s") for word in found if word not in STOP}


def ask(index, question):
    asked = words(question)
    best, position, entry = max(
        (len(asked & words(e.searchable)), -i, e) for i, e in enumerate(index))
    if best < 2:
        return "I could not find that in the handbook."
    return f"{entry.text}\n    [{entry.source}]"

About forty lines, and every call in it was taught earlier: convert_all and status from lesson 29, chunking and contextualize from lessons 23 to 25, where from lesson 27, the word search from lesson 28. A fifth format joins the handbook by adding its name to HANDBOOK, because Docling already returns the same document tree whatever the format was.

This is where the course leads: not a parser demonstration, but the thing a parser is for, a question answered from a shelf of mixed documents with the source shown. From here the search is the weak part, and swapping the word overlap for a real embedding model, indexing the same contextualized strings, is the change that makes it production-worthy.

Try it yourself

Add couriers.html to HANDBOOK and ask which courier serves rural zones.
Ask something not in the handbook and read the fallback.
Print the report from build and check every file is SUCCESS.

PreviousWhat converted, and how well

Every expert started right here.