Doclingdocling 2.127.0 · Python 3.10+
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
26 small wins to finish your pathNext lesson

The headings a chunk carries with it

The second chunk of the shipping document is two bullets about tracking codes. On its own, nothing in it says the word shipping, and a search for shipping will never find it.

Example
from docling.chunking import HierarchicalChunker
from docling.document_converter import DocumentConverter

document = DocumentConverter().convert("shipping.md").document
chunker = HierarchicalChunker()
for chunk in chunker.chunk(document):
    print(repr(chunk.text)[:60])

That is the problem in one line. The text is correct and unfindable, because the words that would have matched are in the headings above it.

The fix is one call

Example
for chunk in chunker.chunk(document):
    print(repr(chunker.contextualize(chunk)))

contextualize puts the headings back on the front. Every chunker has it, and what it produces is the string you index or embed, while chunk.text stays the string you show a person.

Two strings, two jobs

StringUse it forWhy
chunk.textShowing the answerIt is what the document actually says
contextualize(chunk)Searching and embeddingIt carries the words from the headings
chunk.meta.headingsThe citation lineThe same headings, as a list

The handbook in lesson 29 keeps both. Searching the contextualized string is what lets how long does a refund take find a table whose cells say nothing but Card and 5.

Why the headings were even available

Example
for chunk in chunker.chunk(document):
    print(chunk.meta.headings)

The chunker walked up the tree from lesson 7 collecting every heading above each piece. That is also why lesson 16 mattered: on a PDF with every heading flattened to level 1, this list is still produced, but the shape of it no longer tells you which heading was the chapter.

Index the contextualized string, store the plain one. Getting this backwards gives you either a search that cannot find anything or an answer with headings glued to the front of it.
Try it yourself
  • Contextualize the chunks of refunds.html and find the table's headings.
  • Chunk orders.csv and see what meta.headings holds for a file with no headings at all.
  • Print the length of both strings for every chunk and compare.

Slow is fine. Stopping is the only problem.