The contact sheet

A first program built from Part 4: walk the shop's folder, clean every element, and list every email address and phone number it holds, with the files each one came from.

The handbook in lesson 28 never uses the cleaners or the extractors, so this smaller program does. It reads eight files: the six handbook documents plus the FAQ and the support notes from Part 4.

Finding contacts in one element

import re
from unstructured.cleaners.extract import extract_email_address
from unstructured.nlp.patterns import US_PHONE_NUMBERS_PATTERN

PHONE = US_PHONE_NUMBERS_PATTERN.replace(r"\s*$", "")


def contacts_in(element):
    found = extract_email_address(element.text)
    found += [match.group().strip() for match in re.finditer(PHONE, element.text)]
    return found

contacts_in joins the two extractors from lesson 17: extract_email_address as it is, and the phone pattern with its end-of-line anchor removed, so a number in the middle of a sentence is found too.

for element in partition("faq.txt"):
    element.apply(clean_extra_whitespace)
    if contacts_in(element):
        print(contacts_in(element), "|", element.text)

element.apply from lesson 16 tidies the spacing before anything is extracted, which is why both lines print cleanly.

Every file in the folder

FILES = ["shipping.md", "refunds.html", "orders.csv", "ravi.eml",
         "returns.docx", "q3.pptx", "faq.txt", "notes.txt"]

sheet = {}
for name in FILES:
    for element in partition(name):
        element.apply(clean_extra_whitespace)
        for value in contacts_in(element):
            sheet.setdefault(value, set()).add(name)

for value, names in sorted(sheet.items()):
    print(f"{value:22s} {', '.join(sorted(names))}")

A dictionary from each contact to the set of files it appears in, so a value found in three files is listed once. Ravi's address and number turn up in three documents, and the support address in one.

One source is missing. ravi.eml was sent to support@shop.example, yet the sheet does not trace that address to the email. An email's headers are not elements: lesson 14 showed them arriving in the metadata, and contacts_in reads only text.

Reading the email headers too

FILES = ["shipping.md", "refunds.html", "orders.csv", "ravi.eml",
         "returns.docx", "q3.pptx", "faq.txt", "notes.txt"]

sheet = {}
for name in FILES:
    for element in partition(name):
        element.apply(clean_extra_whitespace)
        found = contacts_in(element)
        found += (element.metadata.sent_from or []) + (element.metadata.sent_to or [])
        for value in found:
            sheet.setdefault(value, set()).add(name)
print(sorted(sheet["support@shop.example"]))

sent_from and sent_to are lists on every element of an email, and None on elements from other files, which is what the or [] covers. The support address is now traced to the email as well as the FAQ.