What are the key production metrics for an LLM service?

30-second answerSay your answer out loud first, then reveal.

Latency (time-to-first-token, time per output token or inter-token latency, end-to-end, at p50/p95/p99), throughput (requests/s, tokens/s), errors (rate limits, timeouts, provider errors, parse and validation failures), cost (tokens and $ per request, user, feature, successful outcome), quality (sampled judge scores, user feedback, refusal and abstention rates), safety (guardrail triggers, incidents), and for self-hosted models, resource metrics (GPU utilisation, KV-cache usage, queue length).

Category	Metrics
Latency	TTFT, TPOT/ITL, end-to-end, per-stage (retrieval, tools, LLM), p50/p95/p99
Throughput	RPS, input/output tokens per second, concurrent requests
Reliability	Error rate by type (429, 5xx, timeout), fallback rate, retry rate
Cost	Tokens per request, cost per request, cost per active user, cost per resolved task
Quality	Judge pass rates (groundedness, policy), thumbs up/down, edit and regenerate rates
Safety	Guardrail block rates by category, injection detections, PII leak alerts
Agents	Steps per task, tool error rates, task success rate, loops hit
Self-hosted infra	GPU utilisation, GPU memory, KV-cache utilisation, queue depth, batch size, preemptions

Interview tip: emphasise percentiles over averages, per-stage breakdowns, and cost per outcome rather than per call.

PreviousWhat are the main options for deploying LLMs, and how do you choose?

Little by little, you're building something great.

What are the key production metrics for an LLM service?

Related