chunk_by_title: splitting where the document splits

The second chunker knows what a heading is. It starts a new chunk at every one, so a cut never lands in the middle of a section.

Same file, same size limit, different function.

from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title

chunks = chunk_by_title(partition("shipping.md"), max_characters=200)
for chunk in chunks:
    print(len(chunk.text), "|", chunk.text[:46])

Three chunks instead of two, and every one of them begins with a heading. The Tracking section is whole, its heading and both its bullets in one chunk, which is what a question about tracking needs to match.

Nothing here is about size. All three chunks are well under the limit and the chunker still made three, because the document had three sections.

The surprise

Raise the limit and something unexpected happens.

from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title

chunks = chunk_by_title(partition("shipping.md"), max_characters=400)
print(len(chunks))
print(chunks[0].text[:70])

One chunk containing the whole file. The sections were found and then put back together, which looks like the chunker ignoring its own rule.

It is a second argument doing its job. combine_text_under_n_chars joins a section to the next one when the first is smaller than that number, and it defaults to whatever max_characters is. Every section here is under 400, so every section was combined.

Turning it off

from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title

chunks = chunk_by_title(partition("shipping.md"), max_characters=400,
                        combine_text_under_n_chars=0)
for chunk in chunks:
    print(len(chunk.text), "|", chunk.text[:40])

Four chunks now, one per heading, including a chunk that is the single word Shipping. That is what the argument exists to prevent: a document whose headings were guessed badly, by the ladder in lesson 5, turns into a pile of one-line chunks.

So the setting is a judgement about your documents. Zero when the headings are real, as they are in markdown and HTML and Word. Something near max_characters when they came out of plain text or a PDF and cannot be trusted.

The soft limit

There is a third size argument, and it is the one that gets the two chunkers to agree.

from unstructured.partition.auto import partition
from unstructured.chunking.basic import chunk_elements

chunks = chunk_elements(partition("shipping.md"), max_characters=200,
                        new_after_n_chars=90)
for chunk in chunks:
    print(len(chunk.text), "|", chunk.text[:40])

new_after_n_chars means start a new chunk once this full, and unlike max_characters it can be exceeded, because a chunk is only closed between elements. Three chunks, the same three the title chunker produced, from the chunker that knows nothing about headings.