Searching the chunks

Docling parses documents into chunks, but it does not search them. A handbook that answers a question needs a way to pick the chunk that answers it, and a few lines of word matching does that without pretending to understand meaning.

Words worth comparing

Example

import re

STOP = {"a", "an", "the", "is", "are", "do", "does", "how", "what", "when",
        "who", "i", "my", "of", "to", "for", "it", "and", "can", "you", "your"}


def words(text):
    found = re.findall(r"[a-z0-9]+", text.lower())
    return {word.rstrip("s") for word in found if word not in STOP}


print(sorted(words("How long does a refund take?")))

words lowercases the text, keeps runs of letters and digits, drops a short stop-list of question words, and trims a trailing s so refunds and refund match. What is left is the handful of words that carry the meaning of the question.

Overlap against a chunk

Example

import re

STOP = {"a", "an", "the", "is", "are", "do", "does", "how", "what", "when",
        "who", "i", "my", "of", "to", "for", "it", "and", "can", "you", "your"}


def words(text):
    found = re.findall(r"[a-z0-9]+", text.lower())
    return {word.rstrip("s") for word in found if word not in STOP}


question = words("how long does a refund take?")
chunk_text = "Refunds How long it takes Card, Working days = 5. Bank transfer, Working days = 8"
print(sorted(question & words(chunk_text)))

Three words in common. The chunk that shares the most words with the question is the one to return, and a set intersection counts them. It is not semantic search: it matches words, not meaning, which is why the next line matters.

Search the contextualized text, not the chunk

Example

from docling.document_converter import DocumentConverter
from docling.chunking import HybridChunker

document = DocumentConverter().convert("shipping.md").document
chunker = HybridChunker(max_tokens=64)
for chunk in chunker.chunk(document):
    print(repr(chunk.text[:40]), "<-", repr(chunker.contextualize(chunk)[:40]))

A chunk of two bullet points about tracking codes does not contain the word shipping, so a search for it would miss. contextualize from lesson 24 prepends the headings, Shipping and Tracking among them, so the words a reader would search for are in the text that is searched. The handbook indexes the contextualized string and keeps the plain chunk.text to show as the answer.

Try it yourself

Add shipping to a question and check it now matches the tracking chunk.
Print words for a chunk's plain text and for its contextualized text and compare.
Lower the stop-list by one word and see which questions change.

PreviousThe citation a chunk can build

Slow is fine. Stopping is the only problem.