elements_to_json, and back

A list of elements lasts only as long as the Python process that made it. elements_to_json writes it to a file, and elements_from_json reads it back with nothing lost.

Partitioning a large folder takes time, so a pipeline usually partitions once, saves the result, and works from the saved file afterwards.

Writing elements to a file

Example

import json
from unstructured.partition.auto import partition
from unstructured.staging.base import elements_to_json

elements_to_json(partition("shipping.md"), filename="shipping.json")
data = json.load(open("shipping.json"))
print(len(data))
print(sorted(data[2]))
print(data[2]["type"], "|", data[2]["text"])

Each element became a plain dictionary with four keys. type is the class name from lesson 6, element_id is the id from lesson 10, and metadata holds the fields from lesson 7.

Example

import json
from unstructured.partition.auto import partition
from unstructured.staging.base import elements_to_json

elements_to_json(partition("shipping.md"), filename="shipping.json")
data = json.load(open("shipping.json"))
print(sorted(data[2]["metadata"]))
print(data[2]["metadata"]["parent_id"] == data[1]["element_id"])

parent_id still matches the heading's element_id, so the outline from lesson 8 is in the file too. last_modified is the time the file was last changed on your machine, which is why this course never prints it: your value would not match.

Reading elements back

Example

from unstructured.partition.auto import partition
from unstructured.staging.base import elements_to_json, elements_from_json

elements = partition("shipping.md")
elements_to_json(elements, filename="shipping.json")
back = elements_from_json(filename="shipping.json")
print(type(back[2]).__name__)
print(back[2].id == elements[2].id)
print(back[2].metadata.parent_id == elements[1].id)

They come back as element classes rather than dictionaries, with the same ids and the same links. Code written for a fresh partition works unchanged on elements loaded from a file.

Chunks keep the elements they were made of

Example

import json
from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title
from unstructured.staging.base import elements_to_json, elements_from_json

chunks = chunk_by_title(partition("shipping.md"), max_characters=200)
elements_to_json(chunks, filename="chunks.json")
stored = json.load(open("chunks.json"))[1]["metadata"]["orig_elements"]
print(type(stored).__name__, stored[:20])
again = elements_from_json(filename="chunks.json")
print([element.category for element in again[1].metadata.orig_elements])

In the file, orig_elements from lesson 21 is a compressed string rather than a readable list. elements_from_json unpacks it into elements again, so a citation that relies on it still works after the program restarts.

Unstructured's documentation says the staging functions are being deprecated in favour of the destination connectors in its separate Ingest tools. In version 0.27.6 they still work, and this course uses them because they come with the library itself.

Try it yourself

Call elements_to_json with no filename and pass the string it returns to elements_from_json(text=...).
Open shipping.json and find the parent_id of both Tracking bullets.
Save the chunks of refunds.html and check that the table chunk still has text_as_html after reading it back.

PreviousOverlap, combining, and orig_elements

You understood something today that you didn't yesterday.