LLMOps & Production Interview Questions
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
51 small wins to finish your pathNext question →

Q39HardSystem design

Design a multi-region, highly available LLM-powered service with failover.

30-second answerSay your answer out loud first, then reveal.
Users reach a global load balancer that routes to app tiers in two regions; each region has its own gateway, regional model endpoints and vector index, conversations and configs are replicated, gateways fail over to the other region's endpoints if allowed, and the vector indexes sync by async replication or rebuild.

Design considerations

  1. State: conversation history in a multi-region database (accept a few seconds of replication lag, or pin users to a home region). Vector indexes replicated or rebuilt per region from the same source pipeline.
  2. Model capacity: each region needs quota or capacity for the failover load (an extra region's traffic), not just its normal share. Pre-negotiate provider quotas per region.
  3. Data residency: some tenants can only fail over within allowed regions (e.g. two Indian regions). Routing respects tenant policies.
  4. Degraded modes: if all primary models fail, use a smaller self-hosted model, cached answers, or disable non-critical features.
  5. Testing: regular game days (simulate a region loss); verify RTO/RPO.
  6. Observability: per-region dashboards; global view of failover state; alerts on cross-region traffic shifts.

Trade-offs: cost of duplicate capacity (especially GPUs), consistency vs latency for state, and complexity. Match the design to the business criticality of the feature.

PreviousDesign a continuous improvement pipeline that periodically fine-tunes a model on production feedback.

This is what real progress feels like.