Design the safety and abuse-prevention architecture for a public, free-to-use generative AI app.

30-second answerSay your answer out loud first, then reveal.

Use defence in depth across the whole lifecycle. Account-level controls (sign-up friction, phone or email verification, rate limits, bot detection). Request-level input classifiers (harmful intents, jailbreak patterns, child safety, which is zero tolerance). Model-level alignment and safe system prompts. Output classifiers (text and image). Behavioural monitoring across sessions (abuse patterns, scraping, automated misuse). Reporting and appeals, human review, enforcement actions (warnings, bans), and incident response. Measure both harm prevented and false refusals.

A user request passes an account layer, rate limits and input classifiers (block to refusal), then an aligned model and output classifiers (block to refusal) before the response; refusals and responses feed behavioural monitoring and user reports, which lead to enforcement and classifier retraining.

Key design considerations

Severity tiers: some categories are zero tolerance with legal reporting obligations (e.g. child sexual abuse material), others are context-dependent (medical or security education). Policies define each tier.
Account-level signals catch what single requests can't: many borderline requests in sequence, automated scraping, coordinated misuse.
Latency vs safety: fast classifiers inline; heavier analysis asynchronous (can lead to later account action).
Image/media generation: prompt filtering + output image classifiers + provenance (watermarking, content credentials metadata).
Over-refusal: track false refusal rate on benign test sets; overly strict filters push users away.
Red-teaming: internal and external, before launches and continuously; adversarial test suites in CI.
Privacy: safety logging balanced with data minimisation; access controls for reviewers.
Incident response: playbooks for viral jailbreaks, quick classifier or prompt hotfixes, communication plans.

PreviousYour company debates building its own LLM stack vs buying a vendor platform (or using a single model provider end to end). How do you guide the decision?

Slow is fine. Stopping is the only problem.

Design the safety and abuse-prevention architecture for a public, free-to-use generative AI app.

Key design considerations

Related