Cache and usage: what calls cost

dspy.LM caches responses, so an identical request is answered without calling the provider. track_usage adds token counts to each prediction.

dspy.configure_cache(enable_disk_cache=False)
dspy.configure(lm=lm, track_usage=True)
qa = dspy.Predict("question -> answer")

first = qa(question="Which city is the capital of France?")
print(first.get_lm_usage())
second = qa(question="Which city is the capital of France?")
print(second.get_lm_usage())

configure_cache(enable_disk_cache=False) keeps this example from finding answers saved on disk by an earlier run, so it prints the same every time; the memory cache stays on. track_usage=True makes get_lm_usage() return tokens per model. The mock reports made-up counts, but the shape is what a real provider returns. The second, identical call has no usage: it came from DSPy's cache, and cost nothing.

Where the cache lives

The cache has two layers: memory for the running process, and disk, ~/.dspy_cache by default or the folder in the DSPY_CACHEDIR environment variable. The key is the full request: model, messages and settings such as temperature. Change one word of the prompt, or a demo, and it is a new request.

lm = dspy.LM("openai/gpt-4o-mini", mock_response="[[ ## answer ## ]]\nParis\n\n[[ ## completed ## ]]", cache=False)
dspy.configure(lm=lm, track_usage=True)
qa = dspy.Predict("question -> answer")
qa(question="What is the capital of France?")
print(qa(question="What is the capital of France?").get_lm_usage())

cache=False on the LM sends every call. dspy.configure_cache(enable_disk_cache=False, enable_memory_cache=False) turns caching off for every LM.

When the cache causes surprises

• Re-running an evaluation or optimization after a crash reuses every answer already paid for.
• BestOfN and optimizers deliberately pass a different rollout_id to avoid getting the same cached answer.
• The caching is in dspy.LM. A BaseLM subclass like the stand-in is called every time.