Installation and setup

Promptfoo is a Node command installed from npm. Model-graded checks need a grading model, and a free Gemini key is picked up with no config change.

Node 22 and npm install -g promptfoo

Promptfoo needs Node 22.22 or newer, and its docs recommend Node 24; on an older Node it refuses to start and tells you so. This course uses version 0.123.0.

bash

node --version
npm install -g promptfoo@0.123.0

If you would rather not install anything globally, npx promptfoo@0.123.0 works everywhere this course writes promptfoo.

Example

promptfoo --version

A grading model and hosted providersOptional

This course uses the built-in echo provider, a Python provider and a grader you write in lesson 14, so no key is needed until lesson 22. There, assertions such as llm-rubric call a grading model. Promptfoo picks that grader from whichever key it finds: OpenAI's when OPENAI_API_KEY is set, Gemini's when only GOOGLE_API_KEY is.

Provider	Key	Variable	Provider id
Gemini	Free, aistudio.google.com/apikey	`GOOGLE_API_KEY`	`google:gemini-2.5-flash`
Groq	Free, console.groq.com/keys	`GROQ_API_KEY`	`groq:openai/gpt-oss-120b`
OpenRouter	Free models end in `:free`, openrouter.ai/keys	`OPENROUTER_API_KEY`	`openrouter:google/gemma-4-31b-it:free`
OpenAI	Paid, platform.openai.com/api-keys	`OPENAI_API_KEY`	`openai:gpt-4.1-mini`

export GOOGLE_API_KEY=AIza...
export PROMPTFOO_DISABLE_TELEMETRY=1

The second line turns off promptfoo's anonymous usage data. To grade with Groq or OpenRouter, name the grader in the config:

yaml

defaultTest:
  options:
    provider: groq:openai/gpt-oss-120b

Try it yourself

Run promptfoo --help and find the eval and view commands.

PreviousPromptfoo overview

Little by little, you're building something great.