SimpleDirectoryReader: loading a folder of documents

Real documents live in files. SimpleDirectoryReader turns each file in a folder into a Document, and its metadata travels with every chunk from then on.

The help centre is a folder, help, with three Markdown files:

Examplehelp/refunds.md

# Refunds

You can get a full refund within 30 days of delivery. The money goes back to the card you paid with within 5 working days of us receiving the item.

Items bought in a sale can be refunded too, but the delivery charge is not returned.

To start a refund, open the order in your account and choose Return an item. Print the label and drop the parcel at any post office.

Personalised items cannot be refunded unless they arrive damaged.

The other two follow the same pattern: delivery times and costs, then the shop's two lamps.

Examplehelp/delivery.md

# Delivery

Standard delivery takes 3 to 5 working days and is free on orders over 40.

Express delivery arrives the next working day if you order before 2pm. It costs 6.

We deliver to the mainland only. Parcels to islands take 2 extra working days.

If a parcel has not arrived after 10 working days, contact us and we will send a replacement.

Examplehelp/lamps.md

# Lamps

The LMP-204 desk lamp has a known cable fault. Stop using a lamp with a damaged cable and we will replace it free of charge.

All lamps come with a two year guarantee against electrical faults.

Bulbs are not covered by the refund policy once they have been used.

The LMP-310 floor lamp needs a bulb with an E27 fitting, which is sold separately.

Loading the folder takes one line:

Example

from llama_index.core import SimpleDirectoryReader

documents = SimpleDirectoryReader("help").load_data()
print(len(documents))
print(sorted(documents[0].metadata))

One Document per file. The reader fills in metadata by itself: the file's name, type, size and dates, and its full path on your computer. The loading docs list Markdown, PDF, Word, PowerPoint and more among the formats it reads; for PDFs with tables, the Docling and Unstructured courses parse far better.

Keeping only what you need

Metadata is not just a label: unless a key is excluded, it is included in the text that gets embedded and in the prompt a model sees. The reader already excludes the keys for name, type, size and dates, but not file_path, so the full path from your laptop ends up in both. Give the reader a function that decides the metadata instead:

Example

import os

from llama_index.core import SimpleDirectoryReader


def only_name(path):
    return {"file_name": os.path.basename(path)}


documents = SimpleDirectoryReader("help", file_metadata=only_name).load_data()

print(len(documents))
for document in documents:
    print(document.metadata)

file_metadata is called with each file's path and returns its metadata. Now each document carries its file name and nothing about the machine it was loaded on. The customer data and PII topic makes the same point for real data: decide what travels, before it does.

Try it yourself

Add "folder": os.path.basename(os.path.dirname(path)) to the metadata.
Pass required_exts=[".md"] and add a notes.txt to the folder.
Print documents[0].text[:80].

PreviousVectorStoreIndex: searching documents by meaning

This is what real progress feels like.