The cache, --repeat, and scores that move

Two things make a promptfoo run slower and more expensive than it needs to be, and both have a flag.

The first is asking the same question twice. Promptfoo caches every provider response, so a second run of an unchanged config answers from disk. Here is a provider that takes a second to reply, run twice.

python

import time

def call_api(prompt, options, context):
    time.sleep(1)
    return {"output": "Order A17 shipped on 3 March by courier."}

Example

promptfoo eval > /dev/null; promptfoo eval

The second run reports a duration of zero seconds. Nothing was asked; the answer came from the cache, keyed on the provider and the exact prompt. Change a word of the prompt and the key changes and the provider runs again.

When the cache is wrong

It is wrong whenever you want to know what the provider says now. You changed the model, or the retrieval index, or something on the other side of an API you do not control, and the config did not change. Then the cache hands you yesterday's answer and the run means nothing.

bash

promptfoo eval --no-cache

--no-cache skips it for one run. promptfoo cache clear empties it. The habit worth having is --no-cache in CI and the cache on while you are working on the tests, because while you are editing assertions the answers genuinely have not changed.

The other problem

A real model does not give the same answer twice. Run the suite three times and you may get three pass rates, which makes a single run a poor thing to make a decision on.

bash

promptfoo eval --repeat 3

--repeat runs every test that many times. The interesting output is not the pass rate but the spread: a test that passes three times out of three is stable, and one that passes twice is telling you something about your prompt that a single run hides.

--repeat and the cache fight each other. Repeating a cached call gives you the same answer back three times, which proves nothing. Promptfoo disables the cache when you use --repeat for that reason, and it is the one time you want to pay for the calls.

Where flakiness comes from

Temperature above zero, which is the usual cause and the easiest to rule out.
A rubric that two readers would apply differently, so the judge applies it differently too.
A test whose answer depends on something outside the test, like the time or a live index.
A threshold set exactly at the score the suite normally produces.

Try it yourself

Run the slow provider three times and watch the durations.
Add --repeat 2 to the cached run and see the duration come back.
Run promptfoo cache clear, then the eval again, and compare.

PreviousTwo prompts side by side

Slow is fine. Stopping is the only problem.