Metadata: where an element came from

An element is not only its text. Beside it sits a record of where the text was found, and that record is what makes an answer citable.

Lesson 4 printed it as one line of a dictionary. It is its own object, and it prints as a dictionary of the fields that were actually filled in.

from unstructured.partition.auto import partition

element = partition("shipping.md")[0]
for key, value in element.metadata.to_dict().items():
    print(key, "=", value)

Five fields for a markdown heading: how deep the heading is, what language it is in, the file it came from, what kind of file that was, and when the file was last changed.

The fields are attributes too, which is how you use them in a program.

from unstructured.partition.auto import partition

for name in ["shipping.md", "refunds.html", "orders.csv"]:
    element = partition(name)[0]
    print(element.metadata.filename, "|", element.metadata.filetype)

filetype is the media type the library decided on, not the extension it read. Lesson 10 is about how it decides.

Different files, different fields

The metadata object has about forty fields defined on it and almost none of them apply to any one document. What you get back depends on what the format could tell it.

from unstructured.partition.auto import partition

element = partition("ravi.eml")[0]
for key, value in element.metadata.to_dict().items():
    print(key, "=", value)

The email brought three fields the markdown file could not: who sent it, who it went to, and the subject line. It also brought a last_modified that is not the file's timestamp at all, but the Date header from inside the message.

That is the pattern throughout. A slide deck fills in a page number, a spreadsheet fills in a sheet name, an HTML table fills in text_as_html. Nothing invents a field it cannot fill.

Language

Every element gets a languages list, guessed from the text. You can tell the library instead, which is faster and stops it guessing wrong on short pieces.

from unstructured.partition.auto import partition

elements = partition("notes.txt", languages=["eng"])
print(elements[0].metadata.languages)
print(elements[-1].metadata.languages)

Both say English because we said so. Left to itself the library runs a detector over the whole document once; pass detect_language_per_element=True and it runs on each element instead, which matters for a document that switches language halfway down.