AI System Design Interview Questions

50 questions that come up again and again in AI Engineer, Senior and Staff, and Forward Deployed Engineer design rounds: a framework, the core building blocks, and full designs from ChatGPT-scale chat and Copilot-style completion to voice agents, AI tutors and enterprise AI platforms.

Last updated: 06 Oct, 2026

The questions get harder as you go. The easy questions cover the framework and building blocks every AI design uses: gateways, routing, caching, guardrails, data flywheels, estimation. The intermediate questions cover focused designs and production scenarios: outages, cost spikes, silent quality drift. The hard questions cover full-scale designs and a complete 45-minute mock round.

How to use these questions

30-second answer first. Every question opens with a short answer. Say it out loud before reading on. In a real interview, lead with this, then go deeper if the interviewer asks.
Then the depth. The detailed answer is what a senior interviewer listens for: trade-offs, failure modes, and how you'd actually build it.
Common mistakes. The answers that make interviewers lose interest. Knowing them helps as much as knowing the right answer.
Follow-ups to expect. Interviewers rarely stop at one question. Prepare these and the conversation stays on your ground.

Question types

Type	Questions	What it tests
Concept	19	How things work
Scenario	7	"This broke in production, what do you do?"
System design	24	Whiteboard rounds

Tips for AI system design interviews

Clarify before drawing. Users, use cases, risk of wrong answers, scale and constraints decide the design. Spend the first five minutes here.
Put numbers on it. QPS, tokens per request, cost per day and latency budgets turn opinions into engineering decisions.
Quality is a requirement. Always say how you'll measure output quality: offline evals, online metrics, human review.
Keep the LLM's role bounded. Deterministic code for money, permissions and arithmetic; the LLM for language, extraction and reasoning, with validation.
Design for failure. Provider outages, rate limits, prompt injection and silent drift. Mention fallbacks and monitoring without being asked.

Easy: foundations

#	Question	Type
Q1	How should you structure your answer in an AI system design interview?	Concept
Q2	What are the core building blocks of a production LLM application?	Concept
Q3	What non-functional requirements are specific to AI systems?	Concept
Q4	How do you do back-of-the-envelope estimation for an LLM system?	Concept
Q5	When do you use synchronous, streaming, or asynchronous/batch processing for LLM features?	Concept
Q6	What kinds of caching can you use in LLM systems?	Concept
Q7	What is an LLM gateway, and why do companies build or buy one?	Concept
Q8	What is model routing (or cascading), and how do you design it?	Concept
Q9	How do you handle provider rate limits and quotas in an LLM system?	Concept
Q10	Where do guardrails fit in an AI system, and what do they check?	Concept
Q11	What is a data flywheel for an AI product, and how do you design feedback collection?	Concept
Q12	How should prompts be managed in production?	Concept
Q13	When should you use classical ML instead of an LLM in a system?	Concept
Q14	How do you design systems that tolerate the non-determinism and occasional failures of LLMs?	Concept
Q15	What metrics would you track for an AI product: offline vs online?	Concept

Intermediate: building and debugging

#	Question	Type
Q16	Design a semantic cache for an LLM-powered FAQ and support assistant.	System design
Q17	Design a pipeline that extracts structured data from invoices (PDFs, scans, emails) for an accounts-payable system.	System design
Q18	Design an LLM-assisted content moderation system for a social platform with 50M posts per day.	System design
Q19	Design an AI reply suggestion feature (smart replies) for a customer-support inbox used by 5,000 agents.	System design
Q20	Design a meeting summarisation system: audio → transcript → summary → action items.	System design
Q21	Design a natural-language analytics assistant (text-to-SQL) over a company data warehouse.	System design
Q22	Design AI-powered semantic product search for an e-commerce site with 10M products.	System design
Q23	How do you A/B test LLM features in production?	Concept
Q24	Your primary LLM provider has a major outage during peak hours. How should the system have been designed to handle it?	Scenario
Q25	Traffic will spike 10x during a festival sale. Your AI shopping assistant must stay up. How do you prepare?	Scenario
Q26	A month after launch, the LLM bill is 5x the forecast. How do you investigate and fix it?	Scenario
Q27	Design a batch pipeline to classify and tag 50 million product reviews with an LLM.	System design
Q28	How do you implement token streaming for a chat application end to end?	Concept
Q29	Design conversation storage and memory for a chat product with millions of users.	System design
Q30	Design an AI writing assistant embedded in a document editor (like a "copilot" in Google Docs or Notion).	System design
Q31	How do you protect sensitive data (PII) when using third-party LLM APIs?	Concept
Q32	How do you design an AI system to meet data residency and multi-region requirements?	Concept
Q33	Users say the assistant "got worse" over the last few weeks, but nobody changed anything. How do you investigate, and how do you detect this earlier next time?	Scenario
Q34	Design a real-time voice AI agent (e.g. a phone-based customer support bot).	System design
Q35	How would you use LLMs to improve a recommendation system for a content platform?	System design

Hard: production and design

#	Question	Type
Q36	Design a ChatGPT-like consumer chat application for 10 million daily active users.	System design
Q37	Design an AI code completion system like GitHub Copilot's inline suggestions.	System design
Q38	Design an AI tutor platform that teaches with voice, adapts to each learner, and supports thousands of concurrent users.	System design
Q39	Design an internal enterprise AI platform that 50 product teams use to build LLM features.	System design
Q40	Design a system where a fraud-detection ML model flags transactions and an LLM helps investigators understand and resolve cases.	System design
Q41	Design a customer feedback analytics system that turns millions of reviews, tickets and survey responses into themes and trends.	System design
Q42	Design a self-service fine-tuning platform for internal teams (upload data → train → evaluate → deploy).	System design
Q43	Design an LLM-powered localisation pipeline that translates product content into 20 languages.	System design
Q44	Design a system that sends AI-personalised notifications to 100M users daily.	System design
Q45	Design an AI system to process insurance claims with photos, forms and documents, with human adjusters in the loop.	System design
Q46	The interviewer asks: "Your AI feature must respond in under 500ms at p95. How do you get there?" Walk through your approach.	Scenario
Q47	Your company debates building its own LLM stack vs buying a vendor platform (or using a single model provider end to end). How do you guide the decision?	Scenario
Q48	Design the safety and abuse-prevention architecture for a public, free-to-use generative AI app.	System design
Q49	Design an observability and tracing platform for LLM applications handling billions of spans per day.	System design
Q50	Mock round: "Design an AI assistant for a bank's retail customers." Show how you'd run the 45 minutes.	Scenario

Start: Q1. How should you structure your answer in an AI system design interview?
Interview set: RAG Interview Questions
Interview set: Agentic AI Interview Questions
Interview set: AI Forward Deployed Engineer Interview Questions

Back toInterview prep

Every expert started right here.