Persisting an index: not embedding twice

Embedding thousands of chunks takes time, and money with a hosted model. Persist the index once, and later runs load it instead of rebuilding it.

Example

from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")

import os

from llama_index.core import SimpleDirectoryReader


def only_name(path):
    return {"file_name": os.path.basename(path)}


documents = SimpleDirectoryReader("help", file_metadata=only_name).load_data()

from llama_index.core.node_parser import SentenceSplitter

nodes = SentenceSplitter(chunk_size=80, chunk_overlap=0).get_nodes_from_documents(documents)

from llama_index.core import VectorStoreIndex

index = VectorStoreIndex(nodes)

index.storage_context.persist(persist_dir="storage")
print(sorted(os.listdir("storage")))

storage_context.persist writes the index's parts to a folder as JSON: docstore.json with the chunks' text, index_store.json describing the index, and default__vector_store.json with the embeddings. The graph and image stores are empty files kept for other index types.

Example

from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding

Settings.embed_model = HuggingFaceEmbedding(model_name="sentence-transformers/all-MiniLM-L6-v2")
from llama_index.core import StorageContext, load_index_from_storage

storage = StorageContext.from_defaults(persist_dir="storage")
index = load_index_from_storage(storage)
print(len(index.docstore.docs), "chunks loaded")
print([n.metadata["file_name"] for n in index.as_retriever(similarity_top_k=1).retrieve("How do I get my money back?")])

load_index_from_storage rebuilds the index without re-reading or re-embedding a document. The question is still embedded, with the same model.

Use the same embedding model

The storing guide warns that settings used to build the index, such as the embedding model, must be set the same way when loading. Vectors from one model are meaningless to another, and searching them gives wrong results without any error.

For more than a few thousand chunks or several processes, the storing guide points to a vector database, such as Chroma, Qdrant or PostgreSQL with pgvector, each through its own integration package and the same index interface.

Try it yourself

Delete the storage folder and run the loading code. Read the error.
Time building the index against loading it.
Open storage/docstore.json and find a chunk's text.

PreviousMeasuring retrieval: hit rate on labelled questions

You understood something today that you didn't yesterday.