Design a system that sends AI-personalised notifications to 100M users daily.

30-second answerSay your answer out loud first, then reveal.

Don't call an LLM per notification at send time. Use the classical stack to decide who gets what and when (targeting, ranking, frequency caps, send-time optimisation). Use LLMs offline to generate and vet a library of message templates and variants per segment, intent and language, with slots filled by deterministic code (name, item, price). Optionally use small models to generate per-user text in batch for high-value segments. Enforce brand, safety and compliance checks before anything is sent, and optimise variants with bandits or A/B tests.

Offline, an LLM drafts message variants from a campaign brief that pass automated checks and human approval into a template library; the daily pipeline targets 100M users, applies caps and consent, selects a variant, fills slots deterministically, sends, and feeds engagement back.

Why not per-user LLM generation at send time

100M calls/day means high cost, a latency and throughput bottleneck, and an unreviewable risk (one bad generation goes out to millions).
Personalisation value usually comes from what and when (targeting), more than from bespoke wording.

Where per-user generation can make sense: smaller, high-value segments, run in batch with validation (e.g. weekly personalised learning progress summaries), using templates for structure.

Safety and compliance: no fabricated discounts or claims (prices and offers come from data, never generated), regulatory consent (opt-in), language and cultural review, unsubscribe and complaint monitoring as guardrail metrics.

Metrics: CTR, conversion, unsubscribe rate, complaint rate, long-term retention (avoid notification fatigue).

PreviousDesign an LLM-powered localisation pipeline that translates product content into 20 languages.

This is what real progress feels like.

Design a system that sends AI-personalised notifications to 100M users daily.

Why not per-user LLM generation at send time

Related