Nodes and chunk size: splitting documents

Retrieval returns chunks, not files. How a document is cut decides whether a retrieved chunk holds the whole answer, half of it, or noise.

import os

from llama_index.core import SimpleDirectoryReader


def only_name(path):
    return {"file_name": os.path.basename(path)}


documents = SimpleDirectoryReader("help", file_metadata=only_name).load_data()

from llama_index.core.node_parser import SentenceSplitter

nodes = SentenceSplitter(chunk_size=80, chunk_overlap=0).get_nodes_from_documents(documents)

print(len(documents), "documents ->", len(nodes), "nodes")
for node in nodes:
    print(node.metadata["file_name"], "|", node.get_content().replace("\n", " ")[:60])

A node is a chunk: text, metadata copied from its document, and a link back to it. SentenceSplitter cuts at sentence boundaries where it can, keeping each chunk under chunk_size tokens. chunk_overlap repeats the end of one chunk at the start of the next.

Too large, too small

for size in (60, 400):
    chunks = SentenceSplitter(chunk_size=size, chunk_overlap=0).get_nodes_from_documents(documents)
    refund = [n.get_content().replace("\n", " ") for n in chunks if n.metadata["file_name"] == "refunds.md"]
    print(size, "tokens:", len(chunks), "nodes")
    for text in refund:
        print("   ", text[:90])

At 60 tokens the refund file is two chunks: the refund rules in the first, and the steps for sending an item back split across the end of the first and the start of the second. A question such as how to return an item can only retrieve half of those steps. At 400 each file is one chunk, which is fine for three short files and wasteful for a 50-page manual, where every retrieved chunk would drag pages of unrelated text into the prompt.

There is no correct size, only a trade-off to measure: small chunks match questions precisely and lose context, large ones keep context and match loosely. Lesson 16 measures it.