What are the core building blocks of a production LLM application?

30-second answerSay your answer out loud first, then reveal.

Client/UI → API layer (auth, rate limiting) → orchestration (prompt assembly, workflow or agent logic) → context sources (retrieval, databases, tools, memory) → an LLM gateway (routing, fallbacks, caching, keys, cost tracking) → model providers or self-hosted models. Cutting across all of it: input/output guardrails, observability and tracing, evaluation pipelines, prompt and config management, and a feedback/data store.

Client to API layer to input guardrails to the orchestrator, which uses context sources and the LLM gateway to API or self-hosted models, then output guardrails back to the client; tracing, evals, feedback and a prompt registry sit around it.

What each block is for

Block	Responsibility
API layer	Auth, quotas, session state, request validation
Orchestrator	Builds prompts, calls retrieval and tools, runs workflows or agents, parses outputs
Context sources	RAG indexes, SQL, APIs, user memory
LLM gateway	Single entry point to models: retries, fallbacks, caching, logging, cost attribution
Guardrails	PII redaction, injection/abuse detection, output policy checks, schema validation
Observability	Traces of every step, latency, tokens, cost, errors
Evals + feedback	Offline test suites, online metrics, human labels, improvement loop
Config registry	Versioned prompts, model choices, feature flags

Interview tip. Draw this skeleton early, then say which boxes matter most for this problem and deep-dive those.

PreviousHow should you structure your answer in an AI system design interview?

You understood something today that you didn't yesterday.

What are the core building blocks of a production LLM application?

What each block is for

Related