Chunks that fit

Lesson 22's chunks are whatever size the document made them. An embedding model has a hard limit, and a chunk over it is silently truncated.

HybridChunker starts from the hierarchical chunks and does two more passes: it splits anything too long, then merges neighbours that are too short and share the same headings.

from docling.chunking import HybridChunker
from docling.document_converter import DocumentConverter

document = DocumentConverter().convert("shipping.md").document
chunker = HybridChunker(max_tokens=20)
for chunk in chunker.chunk(document):
    print(repr(chunker.contextualize(chunk))[:70])

Three chunks where lesson 22 gave two. The list of bullets was over twenty tokens once its headings were counted, so it was split, and each half kept the headings.

Counting the tokens

for chunk in chunker.chunk(document):
    text = chunker.contextualize(chunk)
    print(chunker.tokenizer.count_tokens(text), "tokens")

All three are inside the limit, and the count includes the headings because those are part of what gets embedded. Tokens are not words: a token is a piece of a word, and the tokenizer decides where the pieces are.

Whose tokenizer

print(type(chunker.tokenizer).__name__)
print(HybridChunker().tokenizer.get_max_tokens())

With no arguments the chunker uses a Hugging Face tokenizer with a limit of 256, matching a common small embedding model. Downloading that tokenizer is the one thing in this part that touches the network, and it happens once. Pass your own so the count matches the model you are really going to embed with; a limit that does not match is how text goes missing without an error.

The merging pass

Two short paragraphs under the same heading come back as one chunk, because two tiny chunks make two weak search results where one would be strong. Pass merge_peers=False to keep them apart, and repeat_table_header=True, which is the default, to put a wide table's header row on the front of every piece it gets cut into.

print(HybridChunker(max_tokens=20).merge_peers)
print(HybridChunker(max_tokens=20).repeat_table_header)