Changing how a table reads

Lesson 22 turned a table into sentences. That is a choice made by a serializer, and swapping it is how you change any part of the text a chunk contains.

A serializer turns a document, or one piece of it, into text. The exports in part 3 are serializers with friendly names on them. The chunker uses its own set, and they can be replaced one kind at a time.

from docling.chunking import HierarchicalChunker
from docling.document_converter import DocumentConverter

refunds = DocumentConverter().convert("refunds.html").document
for chunk in HierarchicalChunker().chunk(refunds):
    print(repr(chunk.text)[:70])

The default table serializer for chunking writes one statement per cell. It reads well to a model and badly to a person, and for a wide table it is much longer than the grid it replaced.

Swapping the table serializer

Two imports and a four line class. The provider's job is to hand the chunker a serializer for one document; everything except the table part is left alone.

from docling_core.transforms.chunker.hierarchical_chunker import (
    ChunkingDocSerializer,
    ChunkingSerializerProvider,
)
from docling_core.transforms.serializer.markdown import MarkdownTableSerializer

class MarkdownTables(ChunkingSerializerProvider):
    def get_serializer(self, doc):
        return ChunkingDocSerializer(
            doc=doc, table_serializer=MarkdownTableSerializer())

chunker = HierarchicalChunker(serializer_provider=MarkdownTables())
for chunk in chunker.chunk(refunds):
    print(repr(chunk.text)[:70])

The grid is back, inside the chunk. Which one to use depends on what reads the chunk: a model answering questions does better with the sentences, and a model asked to copy a row out of a table does better with the grid.

It works on the hybrid chunker too

from docling.chunking import HybridChunker

hybrid = HybridChunker(max_tokens=64, serializer_provider=MarkdownTables())
for chunk in hybrid.chunk(refunds):
    print(repr(hybrid.contextualize(chunk))[:64])

Same provider, and the token counting from lesson 24 now counts the pipes. That is the trade the grid costs you: a table that fitted in one chunk as sentences may need two as Markdown.

The other serializers

There is one for each kind of piece: text, tables, pictures, lists, inline runs, and a provider wrapping the lot. Subclass any of them the same way. A picture serializer that writes an image's caption instead of a placeholder is four lines, and it is how the enrichment output in lesson 26 reaches a chunk at all.