How do you define SLOs for an LLM-powered service?

30-second answerSay your answer out loud first, then reveal.

Pick a few user-centric service level indicators and set targets over a time window: availability (successful responses, including fallback responses where they're acceptable), latency (e.g. 95% of requests with TTFT under 1.5s), and quality (e.g. ≥ 95% of sampled responses pass the groundedness check weekly). Define error budgets, alert on burn rate, and agree on what happens when the budget is exhausted (freeze risky changes, focus on reliability). Quality SLOs are newer but important for LLM features.

Example SLO set for a support assistant

SLI	SLO (28-day window)
Availability: requests returning a valid response (primary or fallback)	99.9%
Latency: TTFT ≤ 1.5s	95% of requests
Latency: end-to-end ≤ 8s	99% of requests
Quality: sampled responses passing groundedness judge	≥ 95% weekly
Safety: confirmed PII leaks	0

Considerations

Dependency SLOs: your availability is capped by your provider's unless you have fallbacks.
Quality SLIs are statistical: use enough samples and confidence intervals; alert on sustained drops, not single bad responses.
Error budget policy: exhausting the budget triggers a rollback of recent changes and a pause on experiments.
Separate SLOs per feature if criticality differs (customer-facing chat vs internal summarisation).

PreviousWhat are the key production metrics for an LLM service?

You understood something today that you didn't yesterday.

How do you define SLOs for an LLM-powered service?

Example SLO set for a support assistant

Considerations

Related