Failing a build: exit codes and GitHub Actions

An eval nobody runs is a document. The point of the whole exercise is that a change which makes the bot worse cannot be merged, and that means one number: the exit code.

The exit code

Here is the suite with one question whose expected answer is wrong, so a test fails.

promptfoo eval --no-cache > /dev/null; echo "exit code: $?"

Not 1. Promptfoo exits with 100 when at least one test fails, and keeps 1 for real errors like a config it cannot read. That split is deliberate and useful: a build can tell a failing eval apart from a broken setup, and treat them differently.

Two environment variables control it. PROMPTFOO_FAILED_TEST_EXIT_CODE changes the number, which you want when a tool in your pipeline has opinions about exit codes. PROMPTFOO_PASS_RATE_THRESHOLD sets the rate below which the run counts as failed, which is the one you actually want.

A rate, not a count

Demanding that every test passes works for three questions and stops working at forty. One wobbly answer turns the build red, somebody re-runs it, and within a month the job is marked as allowed to fail.

PROMPTFOO_PASS_RATE_THRESHOLD=0.9 promptfoo eval --no-cache

PROMPTFOO_PASS_RATE_THRESHOLD=0.5 promptfoo eval --no-cache > /dev/null; echo "exit code: $?"

Two of three passed, which is above a threshold of 0.5, so the same failing suite now exits zero. Setting that number is a decision about how much regression you are willing to merge, and it belongs to the team rather than to the tool.

The workflow file

Nothing above is specific to GitHub. Any build server that can run a command and read an exit code will do.

name: evals
on: [pull_request]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '24'
      - run: npx promptfoo@latest eval --no-cache -o results.json
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

--no-cache because a cached answer tells you nothing about the change being reviewed. -o results.json because a failure you cannot open is a failure you will rerun. The key goes in as a secret, and in this course there is no key at all, so the step runs as it stands.

What to do with a failure

The useful habit is to keep the results file as a build artifact and read it rather than rerun. promptfoo eval --filter-failing results.json then re-runs only the tests that did not pass, which turns a twenty minute diagnosis into a twenty second one.