Cleaners, and element.apply

Text that came out of a real document is rarely tidy. The library ships about twenty small functions that each fix one thing, and one method that runs them over an element.

Here is the shop's FAQ, saved out of a web page by somebody in a hurry. The spacing survived the trip and the layout did not.

FAQ

Refunds   take   five  working days.

●   Track your order from the email we send.

●   Codes  go live 4 hours after dispatch.

Call the desk on 215-867-5309

Write to support@shop.example or ravi@example.com.

Page 2 of 2

from unstructured.partition.auto import partition

for element in partition("faq.txt"):
    print(f"{element.category:14s} {element.text!r}")

Two things already happened without being asked. The bullet characters are gone, because a partitioner strips the bullet when it decides a line is a ListItem. The runs of spaces are still there, because nothing has been asked to remove them.

One function, one fix

from unstructured.cleaners.core import clean_extra_whitespace

print(repr(clean_extra_whitespace("Refunds   take   five  working days.")))

Every cleaner has that shape: a string in, a string out. clean_extra_whitespace collapses runs of spaces and newlines into single spaces. clean_bullets removes a leading bullet. clean_trailing_punctuation removes a trailing colon or full stop. clean_non_ascii_chars drops anything outside ASCII.

Running one over an element

An element is not a string, so it has a method for this. apply takes any number of cleaners and runs them over the text in order, replacing it.

from unstructured.partition.auto import partition
from unstructured.cleaners.core import clean_extra_whitespace

elements = partition("faq.txt")
for element in elements:
    element.apply(clean_extra_whitespace)
print(repr(elements[1].text))
print(repr(elements[3].text))

The spacing is fixed in both. apply changes the element in place and returns nothing, so there is no new list to keep; the elements you already have are the cleaned ones.

The one with the switches

clean is several of the others behind one call, each turned on by a keyword. They all default to off.

from unstructured.cleaners.core import clean

messy = "  RISK FACTORS ----------  "
print(repr(clean(messy)))
print(repr(clean(messy, extra_whitespace=True, dashes=True, lowercase=True)))

The first call did nothing, because every switch was off and clean with no arguments is a function that strips the ends. The second did three jobs. If a cleaner seems to be doing nothing, that is nearly always why.