Text, markdown and a dataframe

JSON suits a program that reads elements back. The same list can also become plain text, markdown for a language model, or a pandas dataframe for counting and filtering.

Plain text

from unstructured.partition.auto import partition
from unstructured.staging.base import elements_to_text

print(elements_to_text(partition("refunds.html")))

One element per line. The table from lesson 9 became a single line of values, and nothing says which number belongs to which column.

Markdown

from unstructured.partition.auto import partition
from unstructured.staging.base import elements_to_md

print(elements_to_md(partition("refunds.html")))

Two things changed. The table kept its rows and columns, because elements_to_md writes a table's text_as_html into the markdown. The headings lost their levels: the page's h1 and both h2 headings all came out as #.

from unstructured.partition.auto import partition

for element in partition("refunds.html"):
    if element.category == "Title":
        print(element.metadata.category_depth, element.text)

The level is still in the metadata as category_depth. Markdown written by elements_to_md is fine for a model to read, but a program that needs the outline should use the elements, not the markdown.

A pandas dataframe

from unstructured.partition.auto import partition
from unstructured.staging.base import convert_to_dataframe

frame = convert_to_dataframe(partition("refunds.html"))
print(frame[["type", "text"]])
print(frame["type"].value_counts().to_dict())

One row per element and one column per field. Metadata fields become columns of their own, and an element without a value for a field gets an empty cell. Counting elements by type, or finding every table across a folder, is one line of pandas.