Cutting a document into pieces

A document is too big to hand to a search index or a model in one piece. Cutting it up is easy to do badly, and Docling already knows where the joins are.

The usual approach is to export Markdown and cut it every thousand characters. That splits sentences, splits tables down the middle, and throws away every heading. A chunker works on the document object instead, so it cuts on the boundaries Docling already found.

from docling.chunking import HierarchicalChunker
from docling.document_converter import DocumentConverter

document = DocumentConverter().convert("shipping.md").document
for chunk in HierarchicalChunker().chunk(document):
    print(repr(chunk.text))

Two chunks from a six item document, and neither one ends mid sentence. The paragraph is one chunk and the two bullets are another, because a list is one thing and splitting it would leave a bullet with nothing above it.

What a chunk knows

chunk = next(iter(HierarchicalChunker().chunk(document)))
print(chunk.meta.origin.filename)
print(chunk.meta.headings)
print([item.self_ref for item in chunk.meta.doc_items])

The file it came from, the headings it sat under, and the addresses from lesson 5 of the pieces it was built out of. That is a citation, already assembled, and lesson 29 prints exactly these three things.

A table is one chunk

from docling.chunking import HierarchicalChunker
from docling.document_converter import DocumentConverter

refunds = DocumentConverter().convert("refunds.html").document
for chunk in HierarchicalChunker().chunk(refunds):
    print(repr(chunk.text))

The table did not come out as a Markdown table. It came out as sentences: Card, Working days = 5. The chunker writes a table as one short statement per cell, joining each value to its column name, because that reads better to a model than a grid of pipes does. Lesson 25 changes it back if you disagree.

An argument that no longer does anything

The documentation says list items are merged by default and that you can opt out with merge_list_items. In this version that field is deprecated and read nowhere.

plain = [c.text for c in HierarchicalChunker().chunk(document)]
opted = [c.text for c in HierarchicalChunker(merge_list_items=False).chunk(document)]
print(plain == opted)
print(len(plain), "chunks either way")

No error, no warning, no change. It is the second argument in this course that is accepted and ignored, after delim in lesson 10, and the lesson is the same one: check that the setting you changed changed something.