A real model: the swap

Twenty-one lessons, no key. Here is what changes when you use a real one, and it is less than you would think.

Two lines. The provider stops being a file and becomes a model, and the grading provider does the same.

providers:
  - openai:gpt-5
prompts:
  - "Answer from the handbook only. {{question}}"
defaultTest:
  options:
    provider: openai:gpt-5-mini
tests: file://tests.csv

Everything else in the file stays exactly as it is. The tests, the assertions, the metrics, the thresholds, the CSV, the CI job: none of them know or care what answered.

The key

Promptfoo reads keys from the environment: OPENAI_API_KEY, ANTHROPIC_API_KEY and so on, one per provider family. Nothing goes in the config file, which is what makes the config safe to commit.

export OPENAI_API_KEY=sk-...

What changes in practice

It gets slow. The grid from lesson 17 stops being arithmetic and starts being minutes. The cache stops being a nicety.

It costs money. Two providers, two prompts, forty tests and a judge on each is several hundred calls per run, and a judge call is not free either. A cheap model for grading is the usual answer, which is why the config above grades with a smaller model than it tests.

Answers stop repeating. Everything in lesson 18 about --repeat and flakiness starts to matter, and a threshold that sat comfortably at 1.0 will need to come down.

The judge gets better and less predictable. The rubrics you calibrated in lesson 15 should be re-checked against the real grader, because a model judge will disagree with the word-matching one in both directions.

Keep the stand-in

Do not delete it. A local provider and a local judge make a suite that runs on every commit for nothing, which is where the CSV of deterministic checks belongs. Keep the real model for the run that matters, and you get a fast gate and an honest one.