How do you design systems that tolerate the non-determinism and occasional failures of LLMs?

30-second answerSay your answer out loud first, then reveal.

Assume any LLM call can return something malformed, wrong, slow or refused. Use structured outputs with schema validation, retries with repair prompts, timeouts and fallbacks, deterministic code for anything that must be exact (calculations, permissions), idempotent side effects, confidence thresholds with human review, and evaluation with multiple runs. Keep the LLM's role bounded and verifiable.

Patterns

Validate everything: parse JSON against a schema; check business rules (dates valid, amounts within range, IDs exist).
Retry with feedback: send the validation error back to the model ("field 'date' must be ISO format"), limited to N attempts.
Fallback chain: a different model or prompt, a simpler rule-based path, or human review.
Deterministic core: LLM extracts → code computes. Never let the LLM do the final arithmetic on an invoice total, for example.
Low temperature for structured tasks; constrained decoding for formats.
Self-consistency / voting for critical decisions: run 3 times and require agreement.
Timeouts and circuit breakers: don't let one slow call block the pipeline.
Idempotency for any action triggered by LLM output.
Confidence routing: low-confidence items go to a human queue.

Measure: error rates by type (parse failure, validation failure, timeout, refusal) as standard dashboard metrics.

PreviousWhen should you use classical ML instead of an LLM in a system?

This is what real progress feels like.

How do you design systems that tolerate the non-determinism and occasional failures of LLMs?

Patterns

Related