The cache, --repeat, and scores that move
The cache is a local store of provider responses promptfoo keeps between runs, so a repeated call is served from disk instead of made again, and --repeat and --no-cache are how you work with it.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
Promptfoo keeps a cache under your home directory and uses it for responses that cost something to fetch, chiefly calls to a hosted model over the network. This lesson shows where the cache is made, how to switch it off, and how --repeat runs a test more than once. The suite is the single-test config from earlier.
providers:
- file://support_bot.py
prompts:
- "{{question}}"
tests:
- vars:
question: Where is order A17?
assert:
- type: contains
value: 3 March
View the code here
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower() # the rendered prompt is the question
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}
The cache folder is made once
The first run in a fresh home directory prints a line announcing it is creating the cache folder. Run again from the same home and that line is gone, because the folder already exists. Both runs are shown here, one after the other.
# first run: promptfoo makes the cache folder
promptfoo eval --no-table
# second run, same home: the folder is already there
promptfoo eval --no-tableStarting evaluation eval-BVX-2026-10-14T19:18:19 Running 1 test cases (up to 4 at a time)... Creating cache folder at ~/.promptfoo/cache. ✓ Eval complete (ID: eval-BVX-2026-10-14T19:18:19) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 1 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 0s (concurrency: 4) Starting evaluation eval-diX-2026-10-14T19:18:20 Running 1 test cases (up to 4 at a time)... ✓ Eval complete (ID: eval-diX-2026-10-14T19:18:20) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 1 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 0s (concurrency: 4)
Turning the cache off
--no-cache runs without reading or writing the cache, and promptfoo says so before it starts. For the local bot the result is identical; the flag matters when the provider is a model, where it forces a fresh call instead of reusing a stored answer.
# --no-cache skips the cache for this run
promptfoo eval --no-table --no-cacheCache is disabled. Starting evaluation eval-1h3-2026-10-14T19:18:21 Running 1 test cases (up to 4 at a time)... ✓ Eval complete (ID: eval-1h3-2026-10-14T19:18:21) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 1 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 0s (concurrency: 4)
Running a test more than once
--repeat runs every test the given number of times in one eval. Here the one test runs three times and the grid shows three rows. The support bot is deterministic, so all three are identical: its score never moves.
# --repeat runs each test N times; a deterministic bot answers the same each time
promptfoo eval --repeat 3Starting evaluation eval-A35-2026-10-14T19:18:22
Running 3 test cases (up to 4 at a time)...
Creating cache folder at ~/.promptfoo/cache.
┌────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────┐
│ question │ [python:support_bot.py:default] {{question}} │
├────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ Where is order A17? │ [PASS] Order A17 shipped on 3 March by courier. │
├────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ Where is order A17? │ [PASS] Order A17 shipped on 3 March by courier. │
├────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────┤
│ Where is order A17? │ [PASS] Order A17 shipped on 3 March by courier. │
└────────────────────────────────────────────────────────────┴────────────────────────────────────────────────────────────┘
✓ Eval complete (ID: eval-A35-2026-10-14T19:18:22)
» View results: promptfoo view
» Share with your team: https://promptfoo.app
» Feedback: https://promptfoo.dev/feedback
Results:
✓ 3 passed (100%)
0 failed (0%)
0 errors (0%)
Duration: 0s (concurrency: 4)What makes a score move between runs
- The local bot never moves. Same question, same code, same answer, so a repeat run only confirms the rest of the suite held steady.
- A hosted model moves. It can answer differently each call, so its pass rate drifts, and that is what you are measuring when you repeat a test against one.
- The cache can hide the drift. A cached model reply is replayed byte for byte, so repeated runs look stable until you add
--no-cacheto force a real call each time.
What the cache does and does not store
| A hosted model provider | A local file provider | |
|---|---|---|
| Response cached | Yes, keyed on the request | Re-run each time, not reused |
| Why | A network call costs time and money | Running a local function is cheap |
| Effect of --no-cache | Forces a fresh call | No visible change |
Where the cache and --repeat help
- Saving tokens, by not re-paying for a model answer you already have while you iterate on checks.
- Measuring a model's variance, with
--repeatand--no-cachetogether to see how a pass rate drifts. - A clean rerun, with
--no-cachewhen you suspect a stale answer is hiding a real change.
Related
- Previous: Two prompts side by side
- Next: Filtering runs: re-running the failures
- Reference: Promptfoo documentation
- Run the same eval twice from one home and confirm the cache-folder line appears only the first time.
- Run with
--repeat 5and read the five identical rows for the deterministic bot. - Add
--no-cacheand note theCache is disabled.line before the run starts.
Slow is fine. Stopping is the only problem.