Curious builder0 XP earned · 300 to level 2

0 daysFinish a lesson to begin

Badge collection0 of 6 unlocked

29 small wins to finish your path

The shop handbook, end to end

Every piece is written. This lesson puts them together: six files in six formats, chunks that know their section, and answers that name the file and heading they came from.

Loading the folder

python

FILES = ["shipping.md", "refunds.html", "orders.csv",
         "ravi.eml", "returns.docx", "q3.pptx"]


def load(files):
    elements, chunks = {}, []
    for name in files:
        parts = partition(name)
        elements.update({element.id: element for element in parts})
        chunks += chunk_by_title(parts, max_characters=200,
                                 combine_text_under_n_chars=60)
    return elements, chunks

load is the loop from lesson 0 with one addition: it keeps every element in a dictionary by id. partition picks a partitioner per file as in lesson 11, and chunk_by_title from lesson 20 splits where each document's headings are.

Example

elements, chunks = load(FILES)
print(len(elements), "elements,", len(chunks), "chunks")
for chunk in chunks:
    print(chunk.metadata.filename, "|", chunk.text[:40].replace("\n", " "))

Eleven chunks from twenty-three elements. combine_text_under_n_chars joined each short heading to the text under it, so most chunks start with the heading of their section.

Which heading a chunk belongs to

Example

def heading_for(chunk, elements):
    first = chunk.metadata.orig_elements[0]
    if first.category == "Title":
        return first.text
    parent = elements.get(first.metadata.parent_id)
    return parent.text if parent else "no heading"


elements, chunks = load(FILES)
for chunk in chunks:
    print(f"{chunk.metadata.filename:13s} {heading_for(chunk, elements)}")

A chunk that starts with a heading is named after it. A chunk that does not, such as the refunds table, is traced through orig_elements from lesson 21 to its first element, and through that element's parent_id from lesson 8 to the heading above it. That is what the elements dictionary is for. The spreadsheet and the start of the email have no heading to find.

Asking a question

python

def ask(question, chunks, elements, embedder):
    query = embedder.embed_query(question)
    score, chunk = max(((cosine(query, c.embeddings), c) for c in chunks),
                       key=lambda pair: pair[0])
    return round(score, 2), chunk.metadata.filename, heading_for(chunk, elements)

ask scores every chunk against the question and keeps the best. It takes the embedder as an argument, so it never names ShopEmbedder.

Example

elements, chunks = load(FILES)
embedder = ShopEmbedder()
embedder.embed_documents(chunks)
for question in ["are gift cards refundable",
                 "can I return an item after two weeks",
                 "what happens when the courier fails twice"]:
    print(question, "->", ask(question, chunks, elements, embedder))

Each answer names the file and the heading a support agent would open to check it. With a Gemini key, passing GeminiEmbedder() from lesson 25 in place of ShopEmbedder() is the only change the program needs.

handbook.py

Example

import make_office
from unstructured.partition.auto import partition
from unstructured.chunking.title import chunk_by_title
from pretend_unstructured import ShopEmbedder, cosine
FILES = ["shipping.md", "refunds.html", "orders.csv",
         "ravi.eml", "returns.docx", "q3.pptx"]


def load(files):
    elements, chunks = {}, []
    for name in files:
        parts = partition(name)
        elements.update({element.id: element for element in parts})
        chunks += chunk_by_title(parts, max_characters=200,
                                 combine_text_under_n_chars=60)
    return elements, chunks
def heading_for(chunk, elements):
    first = chunk.metadata.orig_elements[0]
    if first.category == "Title":
        return first.text
    parent = elements.get(first.metadata.parent_id)
    return parent.text if parent else "no heading"
def ask(question, chunks, elements, embedder):
    query = embedder.embed_query(question)
    score, chunk = max(((cosine(query, c.embeddings), c) for c in chunks),
                       key=lambda pair: pair[0])
    return round(score, 2), chunk.metadata.filename, heading_for(chunk, elements)


if __name__ == "__main__":
    elements, chunks = load(FILES)
    embedder = ShopEmbedder()
    embedder.embed_documents(chunks)
    score, filename, heading = ask("are gift cards refundable",
                                   chunks, elements, embedder)
    print(f"{filename} > {heading} ({score})")

About forty lines, and every call in it was taught earlier: partitioning in Part 1, metadata and parent_id in Part 2, format detection in Part 3, chunk_by_title and orig_elements in Part 5, and the embedder in Part 6. A seventh file joins by adding its name to FILES, provided the install from lesson 1 includes that format's extra.

Ask it how long does express delivery take and the stand-in returns the email's signature, for the reason lesson 25 showed. That is a limit of counting words into 64 slots, not of the handbook, and it is the question to try first with a real embedding model.

Try it yourself

Add notes.txt to FILES and ask who wants a refund.
Print the top three chunks for a question instead of only the best one.
Save the chunks with elements_to_json from lesson 22 and load them in a second script that only answers questions.

PreviousThe Unstructured API and Ingest

Slow is fine. Stopping is the only problem.