Install unstructured, and partition one file

One function replaces the rules from lesson 2. It takes a path and gives back a list of labelled pieces, and it works the same way whatever the file is.

The library is on PyPI as unstructured. The base package reads plain text, HTML, XML, JSON and email. Everything else is an extra, named after the format it adds.

pip install "unstructured[csv,docx,md,msg,pptx,tsv,xlsx]==0.27.6"

Those seven extras cover the six handbook documents and a few more besides. The version is pinned because a different version would print different things. Lesson 11 shows exactly which formats this install can read and which it cannot.

The first partition

from unstructured.partition.auto import partition

elements = partition("shipping.md")
print(len(elements))

Eight pieces out of a file that the lesson 1 splitter broke into four. The library did not split on headings. It walked the markdown structure and reported every distinct thing it found, which includes each of the two bullets separately.

What the pieces are

from unstructured.partition.auto import partition

elements = partition("shipping.md")
for element in elements:
    print(type(element).__name__, "|", element.text)

Every piece carries a label. Title for the four headings, NarrativeText for the two paragraphs, ListItem for the two bullets. That is the thing the hand-written splitter could not produce: not just the pieces, but what each one is.

The same call, the other two files

from unstructured.partition.auto import partition

for name in ["refunds.html", "orders.csv"]:
    print("==", name)
    for element in partition(name):
        print(" ", type(element).__name__, "|", element.text[:45])

Nothing changed except the filename. The HTML file came back with its two headings and its paragraph labelled, and its table collected into a single Table. The CSV came back as one Table, which is what a CSV file is.

This is the whole promise of the library, and the rest of the course is about what the pieces carry and what to do with them.