Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

Moving the suite to a hosted model

Swapping in a hosted model is a one-line change: the provider line that pointed at a local file now names a model, and the suite around it does not move.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The whole course ran keyless so every page could run with no account. A real suite points the same config at a hosted model. The tests, the asserts and the grading wiring stay exactly as they are; only the provider changes. This page shows that change. It needs an API key to run, so it is shown rather than run, and there is no captured output below.

Swapping the provider under test

The provider line is the only edit. Where it read file://support_bot.py it now names a hosted model, such as openai:gpt-4o-mini. The prompt grows a line of instruction because a real model needs telling what to do, where the local bot answered from a fixed handbook. Everything under tests is unchanged.

yaml
providers:
  - openai:gpt-4o-mini        # was: file://support_bot.py
prompts:
  - "Answer from the shop handbook: {{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March

Swapping the grader too

The A grading script as the stand-in judge lesson wired a Python file as the grader to keep llm-rubric keyless. To grade with a real model, point defaultTest.options.provider at a hosted model the same way. The rubrics, the factuality references and the g-eval criteria do not change; only who reads them does.

yaml
defaultTest:
  options:
    provider: openai:gpt-4o-mini   # was: file://pretend_promptfoo.py

The key it needs

A hosted provider reads its key from the environment, named for the vendor: OPENAI_API_KEY for OpenAI, GROQ_API_KEY for Groq, and so on. With the key set, promptfoo eval runs as before, except each answer and each grade is now a real model call that costs money and varies between runs.

bash
# the hosted provider reads its key from the environment
export OPENAI_API_KEY=sk-...
promptfoo eval
Watch out. A hosted model does not give the same answer every run, so a contains check that passed on the fixed bot may flake on a real one. This is why the model-graded checks exist: a rubric judges whether an answer carries a fact even when the wording moves. Keep the deterministic checks for format and the model-graded ones for meaning.

What stays the same

PieceKeyless courseWith a hosted model
The providerfile://support_bot.py or echoopenai:gpt-4o-mini
The graderfile://pretend_promptfoo.pyA hosted model
The tests and assertsAs writtenUnchanged
The keyNone neededRead from the environment

Where a hosted model belongs

  • The real suite, once the mechanics are learned on the keyless one, pointed at the model you ship.
  • Comparing models, listing two hosted providers to see which answers your tests better.
  • A model-graded check that turns on meaning, where a word-overlap stand-in is too blunt and a real judge is worth the key.
Try it yourself
  • Change the provider line to a model you have a key for and set the matching environment variable.
  • Add a second provider line and compare the two models over the same tests.
  • Point defaultTest.options.provider at a hosted model and keep the rubrics as they are.

This is what real progress feels like.