The Unstructured API and Ingest

This course taught the open source library. The same company offers a hosted API and a separate Ingest tool, and much of what you find online is written for those instead.

The Unstructured API

The library can send a document to Unstructured's hosted API instead of partitioning it locally. partition_via_api does that, and it returns the same kind of elements.

import inspect
from unstructured.partition.api import partition_via_api

parameters = inspect.signature(partition_via_api).parameters
print(parameters["api_url"].default)
print(parameters["api_key"].default == "")

It posts to Unstructured's hosted endpoint unless you pass another api_url, and it needs an api_key from an Unstructured account. The documentation lists what the API has that the library does not:

The documentation describes the open source library as a starting point for quick prototyping, and points production work to Unstructured's hosted pipelines.

Unstructured Ingest

Ingest is a separate project, installed as unstructured-ingest, with a command-line tool and a Python library. It reads batches of files from sources such as cloud storage, runs partitioning, chunking and embedding, and writes the results to destinations such as vector databases. Those destination connectors are what the documentation names as the replacement for the staging functions from lessons 22 and 23.

Two notes from the documentation matter before choosing it. Unstructured recommends its API over Ingest, and says Ingest is not being actively updated with the API's newer features. The old copy inside the library is on its way out as well: importing parts of unstructured prints a warning that unstructured.ingest will be removed and has moved to the unstructured-ingest project.

Formats this course skipped

PDFs and images need the pdf and image extras and several gigabytes of machine-learning packages, and lesson 13 covered the three strategies they use. Formats read through pandoc, such as EPUB, RTF and reStructuredText, need the pandoc program installed outside pip.