Element types, and what decides them

Lesson 3 showed four labels coming out of one file. There are more of them, and the rule that picks one is short enough to read.

Here is a seventh document, the support desk's own scratch notes. It is plain text, which is the hardest case: there is no markup at all to say what anything is.

SUPPORT NOTES

Ravi called about order A17 and said the parcel never arrived at his flat.

He would like a refund.

1. Ask the courier for proof of delivery
2. Offer a refund if there is none

He would like a refund.

Write to ravi@example.com or call 215-867-5309

Thanks!

What comes back

from unstructured.partition.auto import partition

def show(elements):
    for i, element in enumerate(elements):
        print(f"{i:2d} {element.category:14s} {element.text[:48]!r}")

show(partition("notes.txt"))

Seven elements from a file with no formatting in it. The heading became a Title, the sentences became NarrativeText, and the two numbered lines became one ListItem between them.

The interesting one is the last. Thanks! is a sign-off and it came back as a Title, which it plainly is not. That is not a bug to report, it is the rule showing its edge, and the rule is worth knowing.

The rule

For plain text, the library tries a series of questions in order and takes the first one that answers yes. The questions live in unstructured.partition.text_type and you can call them yourself.

from unstructured.partition.text_type import (
    is_possible_narrative_text, is_possible_title)

for line in ["Standard delivery takes 3 working days.",
             "Delivery times", "Thanks!"]:
    print(f"{line!r:44s}",
          is_possible_narrative_text(line), is_possible_title(line))

Two answers for each line. The first line is both a sentence and a plausible title, and it becomes NarrativeText because that question is asked first. The other two are not sentences, so they fall through to the title question, and that question says yes to anything short that does not end in a full stop.

Which is why Thanks! is a Title. Nothing is wrong; the last question in the ladder is a generous one.

The labels you will meet

• Title is a heading, or anything short the ladder could not place better.
• NarrativeText is a paragraph, the thing you usually want to search.
• ListItem is one bullet or one numbered line.
• Table is a whole table, and lesson 8 is about it.
• Text is the label for text that matched nothing.
• EmailAddress, Address, PageBreak and Header turn up in the documents that contain them.

Files with real markup skip the ladder. A markdown ## or an HTML <h2> is a heading because the format says so, which is why shipping.md in lesson 3 came out perfectly and plain text does not.