What does a testing strategy look like for an LLM application?

30-second answerSay your answer out loud first, then reveal.

A pyramid adapted for LLMs. Many fast deterministic unit tests (parsing, prompt building, tool logic, validators) with the LLM mocked. Integration tests with recorded or stubbed LLM responses for the orchestration flows. Contract tests for external tools and APIs. Eval suites running real models on golden datasets (statistical). Safety and red-team tests. Load and performance tests. Production smoke tests and canary analysis. Each layer catches different failures at a different cost.

The testing layers for an LLM application, from many fast unit tests through integration, contract, eval suites and load tests to production checks.

Tips

Mock the LLM for logic tests: a fake that returns scripted outputs (including malformed JSON, refusals, tool calls) to test error handling deterministically.
Record/replay real responses for integration tests to avoid cost and flakiness, and refresh recordings periodically.
Separate flaky statistical evals from deterministic CI checks. Gate on eval thresholds, not exact outputs.
Fault injection: simulate timeouts, 429s and malformed outputs.

PreviousWhat's different about containerising LLM applications and model servers?

Little by little, you're building something great.

What does a testing strategy look like for an LLM application?

Tips

Related