The headings a chunk carries with it
The second chunk of the shipping document is two bullets about tracking codes. On its own, nothing in it says the word shipping, and a search for shipping will never find it.
from docling.chunking import HierarchicalChunker
from docling.document_converter import DocumentConverter
document = DocumentConverter().convert("shipping.md").document
chunker = HierarchicalChunker()
for chunk in chunker.chunk(document):
print(repr(chunk.text)[:60])That is the problem in one line. The text is correct and unfindable, because the words that would have matched are in the headings above it.
The fix is one call
for chunk in chunker.chunk(document):
print(repr(chunker.contextualize(chunk)))contextualize puts the headings back on the front. Every chunker has it, and what it produces is the string you index or embed, while chunk.text stays the string you show a person.
Two strings, two jobs
| String | Use it for | Why |
|---|---|---|
chunk.text | Showing the answer | It is what the document actually says |
contextualize(chunk) | Searching and embedding | It carries the words from the headings |
chunk.meta.headings | The citation line | The same headings, as a list |
The handbook in lesson 29 keeps both. Searching the contextualized string is what lets how long does a refund take find a table whose cells say nothing but Card and 5.
Why the headings were even available
for chunk in chunker.chunk(document):
print(chunk.meta.headings)The chunker walked up the tree from lesson 7 collecting every heading above each piece. That is also why lesson 16 mattered: on a PDF with every heading flattened to level 1, this list is still produced, but the shape of it no longer tells you which heading was the chapter.
- Contextualize the chunks of
refunds.htmland find the table's headings. - Chunk
orders.csvand see whatmeta.headingsholds for a file with no headings at all. - Print the length of both strings for every chunk and compare.
Slow is fine. Stopping is the only problem.