Dropping elements you do not want

A document is full of things nobody wants to search: page numbers, footers, headings that repeat on every page. Dropping them is a filter on the element type.

The FAQ ends with a page number, which the ladder in lesson 5 labelled a Title because it is short and not a sentence.

from unstructured.partition.auto import partition

elements = partition("faq.txt")
print(len(elements))
print(elements[-1].category, "|", elements[-1].text)

Seven elements, and the last one is noise. Fed to a search index it will match questions about page two, which nobody asks.

Filtering by type

from unstructured.partition.auto import partition
from unstructured.staging.base import filter_element_types
from unstructured.documents.elements import Title

elements = partition("faq.txt")
kept = filter_element_types(elements, exclude_element_types=[Title])
for element in kept:
    print(element.category, "|", element.text[:40])

Five elements left, and both of the Titles are gone. That is the trade: dropping headings takes the page number with it and takes the useful heading FAQ as well. Whether it is worth it depends on whether your headings carry meaning, and in the handbook they do, so lesson 27 keeps them.

Two ways to get nothing back

The argument takes classes. Pass the names as strings and the call succeeds and matches nothing.

from unstructured.partition.auto import partition
from unstructured.staging.base import filter_element_types

elements = partition("faq.txt")
print(filter_element_types(elements, include_element_types=["Title"]))

An empty list, no error. The function compares types, a string is never a type, so nothing matches. This is the single easiest way to lose a document silently in a pipeline.

from unstructured.partition.auto import partition
from unstructured.staging.base import filter_element_types
from unstructured.documents.elements import Title, Table

elements = partition("faq.txt")
try:
    filter_element_types(elements, include_element_types=[Title],
                         exclude_element_types=[Table])
except ValueError as error:
    print(error)

One or the other, never both. Say what you want or say what you do not want, and if you need both rules, filter twice.

Or just a comprehension

from unstructured.partition.auto import partition

elements = partition("faq.txt")
kept = [e for e in elements
        if e.category != "Title" or not e.text.startswith("Page ")]
for element in kept:
    print(element.category, "|", element.text[:40])

Six elements: the page number is gone and FAQ stayed. filter_element_types only knows about types, and most real filters are a rule about the text as well. Reach for it when the rule is exactly this type, and write the comprehension when it is not.