Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

The cache, --repeat, and scores that move

The cache is a local store of provider responses promptfoo keeps between runs, so a repeated call is served from disk instead of made again, and --repeat and --no-cache are how you work with it.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

Promptfoo keeps a cache under your home directory and uses it for responses that cost something to fetch, chiefly calls to a hosted model over the network. This lesson shows where the cache is made, how to switch it off, and how --repeat runs a test more than once. The suite is the single-test config from earlier.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

The cache folder is made once

The first run in a fresh home directory prints a line announcing it is creating the cache folder. Run again from the same home and that line is gone, because the folder already exists. Both runs are shown here, one after the other.

Examplethe cache folder is created on the first run only
# first run: promptfoo makes the cache folder
promptfoo eval --no-table

# second run, same home: the folder is already there
promptfoo eval --no-table

Turning the cache off

--no-cache runs without reading or writing the cache, and promptfoo says so before it starts. For the local bot the result is identical; the flag matters when the provider is a model, where it forces a fresh call instead of reusing a stored answer.

Example--no-cache runs without touching the cache
# --no-cache skips the cache for this run
promptfoo eval --no-table --no-cache

Running a test more than once

--repeat runs every test the given number of times in one eval. Here the one test runs three times and the grid shows three rows. The support bot is deterministic, so all three are identical: its score never moves.

Example--repeat runs the test three times, same answer each time
# --repeat runs each test N times; a deterministic bot answers the same each time
promptfoo eval --repeat 3

What makes a score move between runs

  • The local bot never moves. Same question, same code, same answer, so a repeat run only confirms the rest of the suite held steady.
  • A hosted model moves. It can answer differently each call, so its pass rate drifts, and that is what you are measuring when you repeat a test against one.
  • The cache can hide the drift. A cached model reply is replayed byte for byte, so repeated runs look stable until you add --no-cache to force a real call each time.

What the cache does and does not store

A hosted model providerA local file provider
Response cachedYes, keyed on the requestRe-run each time, not reused
WhyA network call costs time and moneyRunning a local function is cheap
Effect of --no-cacheForces a fresh callNo visible change
Watch out. Caching is for provider responses that are expensive to fetch, not for your local Python bot, which promptfoo re-runs on every eval. Edit the bot and rerun with the cache on, and the new answer shows up at once. Do not expect the cache to make a local provider run faster, or to freeze its output.

Where the cache and --repeat help

  • Saving tokens, by not re-paying for a model answer you already have while you iterate on checks.
  • Measuring a model's variance, with --repeat and --no-cache together to see how a pass rate drifts.
  • A clean rerun, with --no-cache when you suspect a stale answer is hiding a real change.
Try it yourself
  • Run the same eval twice from one home and confirm the cache-folder line appears only the first time.
  • Run with --repeat 5 and read the five identical rows for the deterministic bot.
  • Add --no-cache and note the Cache is disabled. line before the run starts.

Slow is fine. Stopping is the only problem.