promptfoo from Node, not the CLI

promptfoo is a Node package as well as a command: import it and call promptfoo.evaluate(config) to run the same grid from your own script and get the results back as an object.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

Every run so far was the promptfoo command reading a YAML file. The same engine is a library you can import. You hand evaluate a config object instead of a file, and it returns the run as a value your code can read, rather than printing a table. This is how you wrap promptfoo in a larger script: generate the tests, run them, and act on the result in one program.

Installing the package

In a project of your own, promptfoo is an ordinary npm dependency, and import promptfoo from 'promptfoo' then resolves from node_modules. This lab ships the package already, so the runnable pages link that installed copy into the working folder, which is the one step a normal install does for you.

# in your own project you install promptfoo as a dependency:
npm install promptfoo@0.124.1

# this lab already has it, so these pages link the installed copy instead:
mkdir -p node_modules && ln -s "$PF_PKG/promptfoo" node_modules/promptfoo

The suite as a JavaScript object

The config is the same shape as the YAML, written as a plain object: a providers list, a prompts list and a tests array where each test carries its vars and its assert list. Here echo is the provider and the second test expects a word the answer does not contain, so the run has one pass and one fail. evaluate is async, so the script awaits it and then reads summary.results and the prompt's metrics.

import promptfoo from 'promptfoo';

// the same suite as the config file, written as a JavaScript object
const config = {
  providers: ['echo'],
  prompts: ['{{question}}'],
  tests: [
    { vars: { question: 'Order A17 shipped on 3 March by courier.' },
      assert: [{ type: 'contains', value: '3 March' }] },
    { vars: { question: 'Refunds take five working days.' },
      assert: [{ type: 'contains', value: 'ten working days' }] },
  ],
};

// evaluate() runs the grid and resolves to the results promptfoo would save
const summary = await promptfoo.evaluate(config, { maxConcurrency: 1 });

for (const r of summary.results) {
  console.log(r.success ? 'PASS' : 'FAIL', '-', r.response.output);
}
const m = summary.prompts[0].metrics;
console.log(`${m.testPassCount} passed, ${m.testFailCount} failed`);

Running the script

With the package linked, Node runs the file directly. Promptfoo prints its own start lines, then the script prints each verdict and the pass and fail counts it read off the returned object.

node check_bot.mjs

Starting evaluation eval-J7K-2026-10-14T19:19:57
Running 2 test cases (up to 1 at a time)...
PASS - Order A17 shipped on 3 March by courier.
FAIL - Refunds take five working days.
1 passed, 1 failed

What the returned object gives you

• summary.results is the list of per-test records, each with a success boolean and the answer under response.output, the same records the JSON output file holds.
• summary.prompts[0].metrics carries the counts, including testPassCount and testFailCount, so your code reads the pass rate without parsing a table.
• The call resolves, it does not exit. Unlike the command, evaluate returns even when a test fails; your script decides what a failure means and what code to exit with.

Where the Node package helps

• Generating tests in code, building the tests array from a database or a file before running them.
• Wrapping the run in a larger tool, where the eval is one step and your code reads the result and moves on.
• A custom report, reading the returned object and writing it out in whatever shape your team wants.

Related

• Previous: Failing a build: exit codes and GitHub Actions
• Next: Sharing, self-hosting and tracing
• Reference: Promptfoo documentation