What you are going to build

The unstructured library turns a document into a list of labelled pieces. A heading becomes a heading, a paragraph becomes a paragraph, a table becomes a table, and each piece remembers which file it came from. This course builds a searchable support handbook out of six documents in six different formats, and every lesson runs on your machine with no API key.

A support desk keeps its answers in whatever file somebody happened to write them in. Shipping rules in a markdown file, refund rules in a web page, open orders in a spreadsheet export, a customer complaint in an email, a returns policy in Word, last quarter's numbers in a slide deck. A program that wants to answer questions from all six has to read all six.

That is the job unstructured does. It is not a PDF reader and it is not a text extractor. It takes a file of almost any kind and gives back the same shape of answer: a list of elements, each with a type, some text, and a record of where it came from.

The handbook, answering a question

import make_office
from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title
from pretend_unstructured import ShopEmbedder, best

chunks = []
for name in ["shipping.md", "refunds.html", "orders.csv",
             "ravi.eml", "returns.docx", "q3.pptx"]:
    chunks += chunk_by_title(partition(name), max_characters=200,
                             combine_text_under_n_chars=60)
ShopEmbedder().embed_documents(chunks)
score, chunk = best(chunks, "are gift cards refundable")
print(round(score, 2), chunk.metadata.filename)
print(chunk.text)

Six files, six formats, one loop. The answer came back with the score, the file it was found in, and the section of that file. Nobody wrote a parser for HTML, or for the Word format, or for the slide deck, and nobody wrote a rule about where a section ends.

That is the program this course builds, in lesson 27. Everything in it is written a piece at a time before then.

Where the course goes

Those four words are the library's own names for its four jobs, and they are the four parts in the middle of this course. Part 1 gets you to the first list of elements. Part 7 builds two programs out of everything before it.

What you need

Python 3.11 or newer, and about two hundred megabytes of disk for the install in lesson 3. No API key, no Docker image and no model download. The six handbook documents are shown in full in the lessons that first read them, and two of them are built by a twelve line script you also see.