Curious builder0 XP earned · 300 to level 2

0 daysFinish a lesson to begin

Badge collection0 of 6 unlocked

51 small wins to finish your pathNext question →

Q18IntermediateSystem design

Design an LLM-assisted content moderation system for a social platform with 50M posts per day.

30-second answerSay your answer out loud first, then reveal.

Use a tiered funnel. Hash matching for known bad content, then fast, cheap classifiers on everything, then an LLM only for the uncertain or high-risk slice (policy reasoning, context, multilingual nuance), then human reviewers for edge cases and appeals. Enforce actions by policy and severity. Continuously retrain the classifiers from LLM and human labels, monitor precision and recall per category and language, and defend against adversarial evasion.

Moderation funnel: 50M posts a day pass hash matching, then fast classifiers that auto-action clear violations and publish the clearly safe ~90%, then LLM policy reasoning on the uncertain ~5%, with ambiguous or severe cases sent to a prioritised human review queue whose labels retrain the classifiers and update the LLM prompts and evals.

Why tiers: 50M posts/day ≈ 580/s on average. Running a frontier LLM on everything is costly and slow, while classifiers are cheap. LLMs add value on nuanced cases: sarcasm, coded language, context-dependent harassment, new policy categories.

LLM component design

Prompt includes the specific policy text, examples, and post context (thread, author history signals).
Structured output: {violation: bool, category, severity, policy_clause, rationale}.
Calibrate against expert labels; separate thresholds per category.

Other considerations

Latency: pre-publication checks must be fast; deeper review can happen after publishing for lower-risk categories.
Multilingual and code-mixed content: evaluate per language; Indic languages and transliterated text are often weak spots.
Adversarial users: misspellings, text in images, emoji codes. Retrain often; use OCR for images.
Fairness: monitor false positive rates across dialects and communities.
Appeals and transparency: decisions are logged with the policy clause for user notices and audits.
Reviewer wellbeing: blur or limit exposure to severe content.

PreviousDesign a pipeline that extracts structured data from invoices (PDFs, scans, emails) for an accounts-payable system.

Slow is fine. Stopping is the only problem.

Design an LLM-assisted content moderation system for a social platform with 50M posts per day.

LLM component design

Other considerations

Related