Testing DSPy programs with DummyLM

Tests should check your program's logic, not a model's mood. DummyLM returns answers you script, in DSPy's format, so tests are fast, free and repeatable.

Example

lm = DummyLM([{"category": "billing"}, {"category": "shipping"}])
with dspy.context(lm=lm):
    sort = dspy.Predict(Triage)
    print(sort(ticket="anything").category)
    print(sort(ticket="anything at all").category)

dspy.utils.DummyLM is DSPy's own test model. Given a list, it answers each call with the next dictionary, formatted with markers so the adapter parses it. It ignores the prompt, so the tests check what your code does with the answers.

Example

lm = DummyLM({"charged twice": {"category": "billing"}, "parcel": {"category": "shipping"}})
with dspy.context(lm=lm):
    sort = dspy.Predict(Triage)
    print(sort(ticket="My parcel never arrived").category)
    print(sort(ticket="I was charged twice").category)

Given a dictionary, it answers with the entry whose key appears in the last message, so the order of calls no longer matters.

A test file

Exampletest_desk.py

import dspy
from dspy.utils import DummyLM

from desk import Desk


def test_sorts_then_replies():
    lm = DummyLM([{"category": "billing"}, {"reply": "Sorry about the double charge."}])
    with dspy.context(lm=lm):
        result = Desk()(ticket="I was charged twice")
    assert result.category == "billing"
    assert result.reply == "Sorry about the double charge."


def test_reply_step_sees_the_category():
    lm = DummyLM([{"category": "shipping"}, {"reply": "Your parcel is on its way."}])
    with dspy.context(lm=lm):
        Desk()(ticket="My parcel never arrived")
    reply_prompt = lm.history[-1]["messages"][-1]["content"]
    assert "[[ ## category ## ]]\nshipping" in reply_prompt

Example

pytest -q

desk.py holds Triage and Desk from lessons 5 and 8. The first test scripts the two calls in order and checks the result. The second reads lm.history to prove that the reply step was given the category the triage step produced, which is the wiring a refactor can break. No key, no network, and the same result every run.

Tests and evals do different jobs

A DummyLM test proves the plumbing. Whether the program sorts tickets well is a question for Evaluate on real data with a real model. You need both.

Try it yourself

Add a test that the Desk skips the reply for account, after making that change from lesson 8.
Script a category of refunds and assert that the call raises.
Use DummyLM as the model in dspy.Evaluate.

PreviousCache and usage: what calls cost

You understood something today that you didn't yesterday.