Design an internal enterprise AI platform that 50 product teams use to build LLM features.

Platform principles

1. Golden paths, not mandates: templates for common patterns (RAG chatbot, extraction workflow, agent with tools) that come with logging, evals and guardrails by default.
2. Central governance:
   • Approved model catalogue with data classification rules (which data can go to which provider).
   • Per-team budgets and chargeback.
   • Security review for agents with write access to systems.
3. Shared retrieval: connectors to enterprise sources with permission-aware indexes, so teams don't each copy sensitive data into their own vector stores.
4. Evaluation as a service: dataset management, LLM-judge templates, CI integration, dashboards per app.
5. Observability: standard tracing (OpenTelemetry GenAI conventions), cost per feature, incident response.
6. Model lifecycle: testing new models across all teams' eval suites before switching defaults; deprecation management.
7. Developer experience: self-service API keys via SSO, sandbox environments, documentation, an internal community of practice.

Success metrics: time-to-first-production feature for a new team, number of apps on the platform, platform-level cost savings (caching, routing), incidents avoided, eval coverage.

Related

• Previous: Q38. Design an AI tutor platform that teaches with voice, adapts to each learner, and supports thousands of concurrent users.
• Next: Q40. Design a system where a fraud-detection ML model flags transactions and an LLM helps investigators understand and resolve cases.
• Model Context Protocol
• DeepEval