How should API keys and secrets be managed in LLM applications?

30-second answerSay your answer out loud first, then reveal.

Store provider API keys and other secrets in a secrets manager (cloud KMS-backed secret stores, Vault), inject them at runtime, never hard-code them in code, prompts, notebooks or client-side apps, use separate keys per environment and service with least-privilege scopes and budgets, rotate them regularly and on suspicion of exposure, centralise provider access through a gateway, monitor usage anomalies, and scan repositories and logs for leaked secrets.

Checklist

No secrets in frontend code. Browser and mobile apps call your backend, which holds the keys. (For bring-your-own-key products, store user keys encrypted and keep them server-side or in secure client storage, never logged.)
Secrets manager + IAM: workloads authenticate with identities (e.g. workload identity), not long-lived keys where possible.
Per-service keys with spend limits, so a compromised key has a bounded blast radius.
Gateway: applications never see provider keys; the gateway holds them.
Rotation: automated rotation; immediate revocation playbook.
Detection: secret scanning in CI and git history; alerts on unusual token usage (a leaked key often shows up as a cost spike).
Never put secrets in prompts: models can be manipulated into revealing system prompts.

PreviousWhat should you log and trace for LLM requests? What is OpenTelemetry's role?

This is what real progress feels like.

How should API keys and secrets be managed in LLM applications?

Checklist

Related