Overlap, combining, and orig_elements

Three things the chunkers do that are not obvious from their output: what they keep about the elements they consumed, what happens to a table, and why the overlap setting appears to do nothing.

Start with what a chunk remembers. Every chunk carries the elements it was made of.

from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title

chunk = chunk_by_title(partition("shipping.md"), max_characters=200)[1]
for element in chunk.metadata.orig_elements:
    print(element.category, "|", element.text[:40])

The heading and both bullets, as the elements they were before they were joined. This is how a citation gets back to the exact element, and how a program can tell a chunk that is one paragraph from a chunk that is a heading plus six bullets.

It is on by default and it costs memory, because the elements are stored twice. include_orig_elements=False turns it off.

Tables are kept apart

from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title

for chunk in chunk_by_title(partition("refunds.html"), max_characters=200):
    print(type(chunk).__name__, len(chunk.text), "|", chunk.text[:40])

Three chunks, and the middle one is still a Table rather than a CompositeElement. A table is never mixed into a chunk of prose, so its text_as_html from lesson 8 survives the trip. That is the isolate_table setting, and it is on by default.

A table larger than the limit is split instead, into TableChunk elements. Only the first of them carries the header row; the rest are rows of values with nothing to say what the columns are, and they are marked is_continuation in their metadata.

The overlap that is not there

Both chunkers take overlap, which should repeat the end of one chunk at the start of the next so a sentence cut in half is still findable. Ask for it.

from unstructured.partition.auto import partition
from unstructured.chunking.basic import chunk_elements

for chunk in chunk_elements(partition("shipping.md"),
                            max_characters=120, overlap=30):
    print(len(chunk.text), "|", chunk.text[:50])

No overlap anywhere. The chunks start exactly where the previous one stopped, and none of them repeats a word.

overlap on its own applies only where the chunker had to cut through the middle of a single oversized element. Between chunks that were split at an element boundary, which is nearly all of them, it does nothing. The argument that applies it everywhere is a second one.

from unstructured.partition.auto import partition
from unstructured.chunking.basic import chunk_elements

for chunk in chunk_elements(partition("shipping.md"), max_characters=120,
                            overlap=30, overlap_all=True):
    print(len(chunk.text), "|", chunk.text[:50])

Six chunks now rather than three, and every one after the first begins in the middle of the sentence that closed the one before it. Thirty characters are repeated each time, which is what most people mean when they ask for overlap, and it takes two arguments rather than one.