An eval with no model

The thing being tested

Checks that need no model

A judge with no key

Comparing

Out of the terminal

A RAG bot, end to end

Learn promptfoo from scratch: config files, providers, assertions, a judge you write yourself, comparing prompts and models, and a CI gate. Every lesson runs with no API key.

Promptfoo

Where the course goes

Promptfoo tutorial: test LLM prompts with no API key

What you are going to build

Promptfoo: why LLM output needs an eval, not a test

Checking answers by hand

Install Promptfoo and write promptfooconfig.yaml

Install promptfoo and run the same check

Promptfoo eval results: pass, fail and the summary

Reading the result table

Promptfoo view: results in the local web viewer

promptfoo view: every run kept

Promptfoo Python provider: test your own app

A Python provider: your own app under test

Promptfoo prompt templates and test vars

Prompts and vars: one prompt, many questions

Load Promptfoo test cases from a CSV file

Tests in a CSV file

Promptfoo deterministic assertions that need no model

contains, equals and regex

Why the schema is the check worth writing

Promptfoo is-json: validate JSON with a schema

is-json: checking the shape of an answer

Write a custom Promptfoo assertion in Python

A Python assertion: a check that is a function

Promptfoo assertion weights, thresholds and named metrics

threshold, weight and named metrics

Promptfoo llm-rubric and choosing the grading provider

llm-rubric: the check that needs a model

You are grading output according to a user-specified rubric. If the statement in the rubric is true, then the output passes the test. You respond with a JSON object with this structure: {reason: string, pass: boolean, score: number}

Examples:

<Output>Hello world</Output>
<Rubric>Content contains a greeting</Rubric>
{"reason": "the content contains the word 'Hello'", "pass": true, "score": 1.0}

<Output>Avast ye swabs, repel the invaders!</Output>
<Rubric>Does not speak like a pirate</Rubric>
{"reason": "'avast ye' is a common pirate term", "pass": false, "score": 0.0}

<Output>
Order A17 shipped on 3 March by courier.
</Output>
<Rubric>
the answer says the order shipped in March
</Rubric>

Write a Promptfoo grading provider with no API key

Writing the stand-in judge

Promptfoo factuality and g-eval assertions, graded locally

factuality and g-eval: two more formats

Calibrating a judge

Promptfoo: calibrate an LLM judge before trusting it

When the judge is wrong

Compare two providers in one Promptfoo eval

Two providers side by side

A/B test prompt wording with Promptfoo

Two prompts side by side

Promptfoo caching and repeat runs for flaky output

The cache, --repeat, and scores that move

Export Promptfoo results as JSON, CSV or JUnit XML

Output files: json, csv and junit

Gate a build on Promptfoo evals in CI

Failing a build: exit codes and GitHub Actions

Point Promptfoo at a real model, not a stand-in

A real model: the swap

Promptfoo context assertions for a RAG bot

Context assertions: faithfulness and recall

Promptfoo red teaming: plugins, strategies and scope

Red teaming: what promptfoo redteam does

The suite, and the lesson each piece came from

Build a Promptfoo eval suite for a RAG support bot

The support bot suite, end to end

A RAG bot, end to end quiz

A retrieval bot scores 1.0 on context-recall and 0.0 on context-faithfulness. What happened?