Element ids, and what changes them

Every element has an id. It is not random, it is not only the text, and knowing what goes into it decides whether you can use it as a key.

Run the same partition twice and the ids are identical.

from unstructured.partition.auto import partition

first = partition("shipping.md")
second = partition("shipping.md")
print([e.id for e in first] == [e.id for e in second])
print(first[0].id)

That is worth relying on. A pipeline can partition a document today and again next week and tell which elements are the ones it has already seen.

What goes into it

The id is a SHA-256 of four things joined together: the filename, the text, the page number and the position of the element on its page. Two identical paragraphs in one file therefore get different ids, because their positions differ.

from unstructured.partition.auto import partition

elements = partition("notes.txt")
print(elements[2].text)
print(elements[4].text)
print(elements[2].id == elements[4].id)

The same sentence twice, two different ids. This is the opposite of what hash of the text would give you, and it is usually what you want: the second mention of a sentence is a different place in the document.

The filename is in there too

from unstructured.partition.auto import partition

with open("shipping.md", "rb") as handle:
    anonymous = partition(file=handle)
named = partition("shipping.md")
print(anonymous[0].text == named[0].text, anonymous[0].id == named[0].id)

Same text, different id, because one of them had no filename to hash. Move a document to a new path and every id in it changes. If you are using ids as database keys, that is the thing to know before the first migration, not after.

Asking for a random id instead

from unstructured.partition.auto import partition

element = partition("shipping.md", unique_element_ids=True)[0]
print(element.id)
print(element.id_to_hash(0))

unique_element_ids=True does not make the hash more unique. It replaces it with a fresh UUID, which is different on every run. id_to_hash converts an element back to the deterministic id, taking the position on the page as its argument.