A persistent vector store with Chroma

Chroma is a vector store that runs inside your own process and writes its index to a folder. The policies are embedded once and read back on every later start instead of rebuilt.

Last updated: 27 Sep, 2026 · LangChain 1.4

Chroma is a vector database that runs inside your own process and writes to a folder. search.py changes at the top; the tool below it is untouched.

Creating a Chroma store

from langchain_chroma import Chroma

store = Chroma(collection_name="policies", embedding_function=WordEmbeddings(),
               persist_directory="./policy_store",           # the folder it writes to
               collection_metadata={"hnsw:space": "cosine"})  # score as cosine, higher is closer

Swapping in the Chroma store

The top of search.py changes: the store becomes a Chroma store, and the chunks are added only when the folder is empty.

from langchain.tools import tool
from langchain_chroma import Chroma
from policies import chunks
from word_embeddings import WordEmbeddings

store = Chroma(collection_name="policies", embedding_function=WordEmbeddings(),
               persist_directory="./policy_store",
               collection_metadata={"hnsw:space": "cosine"})
if not store.get()["ids"]:
    store.add_documents(chunks)

persist_directory is the folder it writes to, and get() returns what is already in there, so the chunks are added once rather than on every start.

The tool, reading relevance scores

The tool below it is untouched, except that it now reads relevance scores from the new store.

@tool
def search_policies(query: str) -> str:
    """Search the shop's policies on refunds, shipping and accounts."""
    found = [doc for doc, score in store.similarity_search_with_relevance_scores(query, k=2)
             if score >= 0.3]
    if not found:
        return "No policy covers this."
    return "\n".join(f"[{doc.metadata['source']}] {doc.page_content}" for doc in found)

The score is not the same number

A store also decides what its scores mean. Chroma's default answers with a distance, where lower is closer, and a plain similarity_search_with_score shows it.

from langchain_chroma import Chroma
from policies import chunks
from word_embeddings import WordEmbeddings

plain = Chroma(collection_name="plain-store", embedding_function=WordEmbeddings())
plain.add_documents(chunks)
for doc, score in plain.similarity_search_with_score("do you sell gift cards", k=2):
    print(round(score, 2), doc.metadata["source"])

Those are distances: 11 and 12, for a question no policy covers. A score >= 0.3 cut would have let both through and the desk would have quoted the shipping policy at a customer asking about gift cards. Two changes keep the old behaviour: hnsw:space asks for cosine, and similarity_search_with_relevance_scores returns 0 to 1 with higher meaning closer.

The desk on the real store

from chat import say
from search import store

say("ravi", "How long does a refund take?", "ravi-9")
say("ravi", "Do you sell gift cards?", "ravi-9")
print(len(store.get()["ids"]), "chunks kept in ./policy_store")

The refund policy is found and the gift card question is still refused, so the cut survived the move. The five chunks now sit in ./policy_store, and the next start reads them instead of splitting the documents again.

What the real store changed

• Distances, not similarities. Chroma's default score is a distance where lower is closer, so 11 and 12 mean the gift-card question matched nothing well.
• Two lines keep the old cut. hnsw:space asks for cosine and similarity_search_with_relevance_scores returns 0 to 1 with higher meaning closer, so score >= 0.3 means the same thing again.
• The index survives the exit. Five chunks stay in ./policy_store, and the next start reads them instead of splitting the documents again.

The embedder is the same swap

WordEmbeddings is two methods, and a hosted embedder is the same two. Installing langchain-openai and changing the argument is the whole move.

from langchain_openai import OpenAIEmbeddings

store = Chroma(collection_name="policies", embedding_function=OpenAIEmbeddings(),
               persist_directory="./policy_store",
               collection_metadata={"hnsw:space": "cosine"})

It reads OPENAI_API_KEY and charges for what it embeds. One thing does change, though. Vectors from one embedder mean nothing to another, so the folder has to be deleted and the chunks added again, and the 0.3 cut is worth re-checking against real vectors, which sit much closer together than word counts do.

When policies live on disk

• Keeping an embedded policy set on disk so a restart does not rebuild it.
• Moving from the in-memory store to a database that other processes can read.

Related

• Previous: The desk on a hosted model
• Next: Saving threads with a SqliteSaver checkpointer
• Reference: Retrieval and vector stores