Design the guardrails architecture for a bank's customer chatbot with under 300ms of added latency.

30-second answerSay your answer out loud first, then reveal.

Use tiered, mostly parallel checks. Fast deterministic checks inline (auth context, PII regexes, length, allow-listed intents), small, fast classifiers for injection, abuse and topic running in parallel with retrieval, and strict action guardrails in code (authorisation, limits, confirmations) that add no LLM latency. On output, apply chunk-level PII and policy checks on the stream, with heavier groundedness checks only for high-risk answer types or asynchronously. Fail closed for money-moving actions.

A bank chatbot message passes inline deterministic checks, then injection and topic classifiers run in parallel with retrieval into a decision that returns a template response or streams LLM generation through chunk-level output checks, with tool requests going to action guardrails in code and async groundedness judging after the user sees the answer.

Latency budget (illustrative)

Check	Placement	Added latency
Deterministic input checks	Inline	~5ms
Injection + topic classifiers	Parallel with retrieval	~0 extra (hidden behind retrieval)
Output chunk checks	Streaming	~10–30ms per chunk
Action guardrails	Code at tool boundary	Negligible
Groundedness judge	High-risk answers only (rates/fees) or async	Selective

Design principles

Fail closed for actions (transfers, card changes); fail open with logging for low-risk informational guardrail outages (documented decision).
Self-hosted small classifiers near the app for low latency and data residency.
Template answers for sensitive topics (fraud reporting steps, complaints) instead of free generation.
Measure the false positive and false negative rates of each guardrail, plus p95 added latency.

PreviousDesign online evaluation and quality monitoring for an LLM application in production.

Slow is fine. Stopping is the only problem.

Design the guardrails architecture for a bank's customer chatbot with under 300ms of added latency.

Latency budget (illustrative)

Design principles

Related