The interviewer asks: "Your AI feature must respond in under 500ms at p95. How do you get there?" Walk through your approach.

30-second answerSay your answer out loud first, then reveal.

First check that it's feasible: measure the current latency breakdown per stage and decide what "respond" means (first token, or the complete answer?). Then cut every stage: a smaller or fine-tuned model, shorter prompts and outputs, prompt caching, co-located and warm infrastructure, parallel retrieval, precomputation and caching, streaming the first meaningful output, and removing sequential LLM calls (no agents in the hot path). Manage tail latency with timeouts, hedged requests and capacity headroom. Change the UX if needed.

Step 1 — Clarify: first token vs complete response? What output length? Which percentile, and measured where (server vs client)?

Step 2 — Measure the breakdown (example before)

Stage	p95
Network + auth	60ms
Retrieval (embed + search)	120ms
LLM TTFT (large model, 3K-token prompt)	900ms
Generation (150 tokens)	2,000ms

Step 3 — Levers

Model: a small model (fine-tuned or distilled for the task) self-hosted near the app; quantized (FP8/INT4); speculative decoding.
Prompt: cut tokens (shorter instructions, fewer chunks); prompt caching for the static prefix; avoid long histories.
Output: constrain length (labels, short answers); structured outputs; generate only what's needed.
Pipeline: remove sequential LLM calls (no separate rewrite step, or use a tiny model for it); run retrieval in parallel with other work; precompute embeddings for known queries.
Caching: exact and semantic caches for frequent requests; precomputed answers for the head of the distribution.
Infrastructure: warm GPU capacity, low queueing (headroom, admission control), persistent connections, same-region deployment.
Tail latency: timeouts with fallback (cached or simpler answer), hedged requests to a second replica for slow ones, isolation from batch workloads.
UX: stream the first token within 500ms and fill in the rest; optimistic UI; precompute when user intent is predictable (prefetch on hover or focus).

Step 4 — Verify: load test at peak, track p95/p99 continuously, and add latency regression checks to CI.

PreviousDesign an AI system to process insurance claims with photos, forms and documents, with human adjusters in the loop.

Little by little, you're building something great.

The interviewer asks: "Your AI feature must respond in under 500ms at p95. How do you get there?" Walk through your approach.

Step 2 — Measure the breakdown (example before)

Step 3 — Levers

Related