Failing a build: exit codes and GitHub Actions
promptfoo eval exits 100 when a test fails, and that non-zero code is the whole point of running it in a build: a build job reads the exit code and stops when the suite goes red.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
A failed test in the Reading the result table lesson still printed its grid and scrolled away. In a build there is no one reading the grid; the build job reads the exit code. This lesson shows the code a failed run returns, two environment variables that change it, and the shape of a GitHub Actions step that runs the suite.
The exit code of a failed run
The mixed suite from the last lesson has one passing test and one failing one. Running it and printing $? shows the exact code. A passing run returns 0; this run returns 100, a number chosen so a failed eval is hard to confuse with an ordinary error.
providers:
- file://support_bot.py
prompts:
- "{{question}}"
tests:
- vars:
question: Where is order A17?
assert:
- type: contains
value: 3 March
- vars:
question: Where is order B99?
assert:
- type: contains
value: shipped
View the code here
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower() # the rendered prompt is the question
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}
# a failed test makes eval exit 100, not 1
promptfoo eval --no-table
echo "exit code: $?"Starting evaluation eval-eXE-2026-10-14T19:19:53 Running 2 test cases (up to 4 at a time)... Creating cache folder at ~/.promptfoo/cache. ✓ Eval complete (ID: eval-eXE-2026-10-14T19:19:53) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 1 passed (50.00%) ✗ 1 failed (50.00%) 0 errors (0%) Duration: 0s (concurrency: 4) exit code: 100
Changing the code with two variables
Two environment variables change what a run returns. PROMPTFOO_FAILED_TEST_EXIT_CODE sets the number a failed test exits with, for a build that expects a particular code. PROMPTFOO_PASS_RATE_THRESHOLD turns the gate into a pass rate: the run exits 0 as long as the pass rate meets the bar, and non-zero when it falls below, so a suite with a known-flaky case can still go green above a line you set.
# change the number a failed test exits with
PROMPTFOO_FAILED_TEST_EXIT_CODE=7 promptfoo eval --no-table >/dev/null 2>&1
echo "failed-test code: $?"
# or gate on a pass rate instead: 50% clears a 50 bar, not a 90 one
PROMPTFOO_PASS_RATE_THRESHOLD=50 promptfoo eval --no-table >/dev/null 2>&1
echo "threshold 50: $?"
PROMPTFOO_PASS_RATE_THRESHOLD=90 promptfoo eval --no-table >/dev/null 2>&1
echo "threshold 90: $?"failed-test code: 7 threshold 50: 0 threshold 90: 100
- The override exits 7, the number handed to
PROMPTFOO_FAILED_TEST_EXIT_CODE, in place of the default 100. - A 50% pass rate clears a 50 bar and the run exits 0, because the rate meets the threshold.
- The same 50% fails a 90 bar and the run exits 100, because the rate is below it.
Running the suite in GitHub Actions
In a workflow the eval is one step after the code is checked out and Node is set up. No other wiring is needed: when a test fails the step exits non-zero and the job turns red. This block belongs in a repository under .github/workflows/, so it is shown rather than run here.
# .github/workflows/evals.yml
name: evals
on: [push]
jobs:
promptfoo:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: "24"
- name: Run promptfoo evals
run: npx promptfoo@0.124.1 eval
The step pins the version with npx promptfoo@0.124.1 so a later release cannot change the suite under you, and it needs no API key because every provider and grader in this course runs without one.
PROMPTFOO_FAILED_TEST_EXIT_CODE rather than guessing the default.Where a build gate helps
- A pull request check, where the eval runs on every change and blocks a merge that drops an answer.
- A nightly run, where a pass-rate threshold lets a known-flaky case through while catching a real regression.
- A release step, where the suite must be green before the build is allowed to ship.
Related
- Previous: Output files: json, csv and junit
- Next: promptfoo from Node, not the CLI
- Reference: Promptfoo documentation
- Fix the failing test so the suite passes, rerun, and confirm the exit code is 0.
- Set
PROMPTFOO_PASS_RATE_THRESHOLD=50on the failing suite and read the exit code. - Add the
-o out.junit.xmlflag to the GitHub Actions step so the build shows each test.
Little by little, you're building something great.