Why elements are not chunks

Elements are the shape the document had. Chunks are the shape a search index wants. They are not the same shape, and the gap is what chunking closes.

The eight elements of shipping.md include four headings. A heading on its own is three words with no answer in it, and a bullet on its own has lost the heading that said what it was about.

from unstructured.partition.auto import partition

for element in partition("shipping.md"):
    print(len(element.text), element.category, "|", element.text[:36])

Eight rows, four of them under twenty characters. Put those in a search index and a question about tracking matches the word Tracking, which tells the reader nothing they did not already ask.

Packing them back together

The simplest chunker fills a chunk to a size and starts another.

from unstructured.partition.auto import partition
from unstructured.chunking.basic import chunk_elements

chunks = chunk_elements(partition("shipping.md"), max_characters=200)
for chunk in chunks:
    print(type(chunk).__name__, len(chunk.text), "|", chunk.text[:44])

Two chunks, each under two hundred characters, each holding several elements joined by blank lines. The type changed: a chunk of joined text is a CompositeElement, which is an element like any other and has text and metadata in the same places.

The default for max_characters is 500. It is a hard limit, and it is the only one of the chunking arguments that has to be obeyed.

What the size cut ignores

from unstructured.partition.auto import partition
from unstructured.chunking.basic import chunk_elements

chunks = chunk_elements(partition("shipping.md"), max_characters=200)
print(chunks[0].text[-60:])
print("---")
print(chunks[1].text[:60])

The cut landed in the middle of the Tracking section. One bullet went into the first chunk and the other into the second, and the heading that explains them both is in the first. A question about tracking codes will now match half of the answer.

That is the whole argument for the second chunker, and it is lesson 19.

A chunk is still an element

from unstructured.partition.auto import partition
from unstructured.chunking.basic import chunk_elements

chunk = chunk_elements(partition("shipping.md"), max_characters=200)[0]
print(chunk.metadata.filename, chunk.metadata.filetype)
print(chunk.id[:8])

Filename, file type and an id, exactly as in lesson 6 and lesson 9. Everything the elements knew about where they came from survives into the chunk, which is what makes a citation possible after chunking.