Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

Failing a build: exit codes and GitHub Actions

promptfoo eval exits 100 when a test fails, and that non-zero code is the whole point of running it in a build: a build job reads the exit code and stops when the suite goes red.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

A failed test in the Reading the result table lesson still printed its grid and scrolled away. In a build there is no one reading the grid; the build job reads the exit code. This lesson shows the code a failed run returns, two environment variables that change it, and the shape of a GitHub Actions step that runs the suite.

The exit code of a failed run

The mixed suite from the last lesson has one passing test and one failing one. Running it and printing $? shows the exact code. A passing run returns 0; this run returns 100, a number chosen so a failed eval is hard to confuse with an ordinary error.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March
  - vars:
      question: Where is order B99?
    assert:
      - type: contains
        value: shipped
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}
Examplea failing suite exits 100
# a failed test makes eval exit 100, not 1
promptfoo eval --no-table
echo "exit code: $?"

Changing the code with two variables

Two environment variables change what a run returns. PROMPTFOO_FAILED_TEST_EXIT_CODE sets the number a failed test exits with, for a build that expects a particular code. PROMPTFOO_PASS_RATE_THRESHOLD turns the gate into a pass rate: the run exits 0 as long as the pass rate meets the bar, and non-zero when it falls below, so a suite with a known-flaky case can still go green above a line you set.

Examplean exit code you choose, or a pass-rate gate
# change the number a failed test exits with
PROMPTFOO_FAILED_TEST_EXIT_CODE=7 promptfoo eval --no-table >/dev/null 2>&1
echo "failed-test code: $?"

# or gate on a pass rate instead: 50% clears a 50 bar, not a 90 one
PROMPTFOO_PASS_RATE_THRESHOLD=50 promptfoo eval --no-table >/dev/null 2>&1
echo "threshold 50: $?"
PROMPTFOO_PASS_RATE_THRESHOLD=90 promptfoo eval --no-table >/dev/null 2>&1
echo "threshold 90: $?"
  • The override exits 7, the number handed to PROMPTFOO_FAILED_TEST_EXIT_CODE, in place of the default 100.
  • A 50% pass rate clears a 50 bar and the run exits 0, because the rate meets the threshold.
  • The same 50% fails a 90 bar and the run exits 100, because the rate is below it.

Running the suite in GitHub Actions

In a workflow the eval is one step after the code is checked out and Node is set up. No other wiring is needed: when a test fails the step exits non-zero and the job turns red. This block belongs in a repository under .github/workflows/, so it is shown rather than run here.

yaml
# .github/workflows/evals.yml
name: evals
on: [push]
jobs:
  promptfoo:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "24"
      - name: Run promptfoo evals
        run: npx promptfoo@0.124.1 eval

The step pins the version with npx promptfoo@0.124.1 so a later release cannot change the suite under you, and it needs no API key because every provider and grader in this course runs without one.

Watch out. A failed test exits 100, not 1. A build script that treats any non-zero code the same will still fail correctly, but one that checks for an exit code of exactly 1 will miss it. If you need a specific code, set PROMPTFOO_FAILED_TEST_EXIT_CODE rather than guessing the default.

Where a build gate helps

  • A pull request check, where the eval runs on every change and blocks a merge that drops an answer.
  • A nightly run, where a pass-rate threshold lets a known-flaky case through while catching a real regression.
  • A release step, where the suite must be green before the build is allowed to ship.
Try it yourself
  • Fix the failing test so the suite passes, rerun, and confirm the exit code is 0.
  • Set PROMPTFOO_PASS_RATE_THRESHOLD=50 on the failing suite and read the exit code.
  • Add the -o out.junit.xml flag to the GitHub Actions step so the build shows each test.

Little by little, you're building something great.