How do you keep a domain-specific bot on topic (scope control)?

30-second answerSay your answer out loud first, then reveal.

Combine a clear scope definition in the system prompt (what the bot does and doesn't do, and how to respond to out-of-scope requests), an intent or topic classifier that detects off-topic requests before the main model (cheap and controllable), retrieval-based grounding (answer only from domain content, abstain otherwise), and graceful out-of-scope responses that redirect users. Evaluate with in-scope, out-of-scope and borderline test sets.

Layers

Prompt policy:
• "You help customers of Acme Bank with accounts, cards and loans. For other topics, say briefly that you can't help with that and suggest what you can do."
Topic classifier (small model or LLM call): in-scope / out-of-scope / harmful. Off-topic → template response without calling the expensive model.
Grounding: require retrieved support; if retrieval finds nothing relevant, abstain.
Output check: detect drift into unrelated content (e.g. writing poems, coding help) for strictly scoped bots.

Borderline handling: "Should I invest in mutual funds?" may be in scope for product information but out of scope for personalised financial advice. Define these boundaries explicitly with examples.

Metrics: out-of-scope detection recall (users trying to use the bot as a free general-purpose chatbot costs money and creates brand risk), in-scope false rejection rate, and user satisfaction with redirections.

Why it matters: cost control (abuse of free LLM access), brand safety (screenshots of the bot saying odd things), and legal risk (advice outside licensed scope).

PreviousCompare guardrail approaches: NeMo Guardrails, Guardrails AI, safety classifier models, and custom code.

Little by little, you're building something great.

How do you keep a domain-specific bot on topic (scope control)?

Layers

Related