What is an LLM gateway, and why do companies build or buy one?

30-second answerSay your answer out loud first, then reveal.

An LLM gateway is a central proxy between applications and model providers. It offers a unified API across models, plus routing, retries and fallbacks, rate limiting and quotas per team, key management, caching, logging and tracing, cost attribution, and policy enforcement (PII filtering, allowed models). It gives an organisation control, visibility and flexibility, so changing models becomes configuration rather than code.

Three apps call one LLM gateway, which holds auth and team quotas, routing and fallbacks, a cache, PII and allowed-model policy, and logging with cost attribution, in front of Provider A, Provider B and self-hosted vLLM.

Key capabilities

Unified interface (often OpenAI-compatible) across providers.
Reliability: retries with backoff, failover to another provider or region, circuit breakers, timeouts.
Governance: which teams can use which models; data policies (no sensitive data to certain providers).
Cost control: per-team budgets, alerts, chargeback reports.
Observability: a central log of prompts and responses (with redaction), latency, errors.
Caching and routing (Q6, Q8).
Secrets: apps never hold provider keys directly.

Options: open-source (LiteLLM, Portkey, Kong / Envoy AI gateway plugins), cloud-native gateways, or in-house.

Trade-offs: an extra network hop (a few ms), a single point of failure (so make it highly available), and a feature lag when providers ship new API features.

PreviousWhat kinds of caching can you use in LLM systems?

You understood something today that you didn't yesterday.

What is an LLM gateway, and why do companies build or buy one?

Key capabilities

Related