Testing that memory works

A memory layer fails quietly. Nothing crashes when a fact is not stored or not found; the assistant simply answers as though it never knew, which looks like a model problem and is not.

So it is worth testing, and the stand-ins make that cheap: no key, no network, and the same answer every run. The three checks below use the shop.py from the last lesson.

The three things worth asserting

Example

from shop import assistant, memory

shop = memory()
assistant(shop, "ravi", "I prefer email updates, not SMS.")
found = shop.search("email updates", filters={"user_id": "ravi"})["results"]

print("stored something ", len(shop.get_all(filters={"user_id": "ravi"})["results"]) > 0)
print("found it again   ", len(found) > 0)
print("kept it separate ", shop.get_all(filters={"user_id": "priya"})["results"] == [])

It was stored. The commonest failure, and the one with no symptom: an extraction that returns nothing leaves an empty result and no error, exactly as in lesson 4.

It can be found. Storing and retrieving are different mechanisms, and a memory that is stored but unreachable is no better than one that was dropped.

It did not leak. One customer's memory appearing under another is the failure that matters most, and the only one of the three that is a real incident rather than a disappointment.

The leak test is the one to write first

Example

from shop import assistant, memory

shop = memory()
assistant(shop, "ravi", "Deliver to my office.")
assistant(shop, "priya", "Deliver to my home.")

for user in ("ravi", "priya"):
    found = shop.search("deliver", filters={"user_id": user})["results"]
    print(user, "->", [hit["memory"] for hit in found])

Two customers, two answers, no crossing over. It passes because every call carries an identifier, and it is worth an automated check precisely because it will keep passing right up until somebody adds a code path that forgets one.

Where this fits

These are ordinary assertions and they belong in whatever you already run. The site has three courses on evaluating the model's side of this: DeepEval and RAGAS for scoring answers, and Promptfoo for running a suite of questions from a file and failing a build. A memory layer is the input to all of them.

Test the memory layer without the model, and the model without the memory layer. When an assistant gives a bad answer, either it did not know the fact or it knew and ignored it. The assertions above answer the first half in milliseconds, which tells you whether to look at the second.

Try it yourself

Delete the filters from one search and see what comes back.
Rewrite the three checks as assert statements and run them under pytest.
Add a memory with no user_id and confirm Mem0 refuses it.

PreviousPutting memory in the reply loop

You understood something today that you didn't yesterday.