How do dev, staging and production environments work for LLM applications? What's tricky about them?

30-second answerSay your answer out loud first, then reveal.

As in normal software: dev for building, staging for integration and evals, production for users. The tricky parts are data (staging needs realistic but privacy-safe documents and indexes), external model dependencies (the same model versions and provider settings across environments, separate API keys and quotas), cost (eval runs in staging consume real tokens), and non-determinism (staging tests are statistical, not exact). Keep configs identical except for clearly declared differences.

Practices

Concern	Approach
Model parity	Same pinned model versions in staging and prod; separate keys and quotas per environment
Data	Staging indexes built from anonymised or synthetic copies; refreshed regularly
Secrets	Separate secrets per environment; never share production keys
Cost control	Budgets per environment; cheaper models for dev iteration, but final evals on the production model
Tools / integrations	Sandbox accounts of external systems (CRM sandbox, test payment gateway)
Evals	Run the full eval suite in staging before promotion; smoke evals after production deploy
Config drift	Infrastructure-as-code and a single config source with environment overlays

Production-only effects to plan for: real traffic distribution, provider rate limits at scale, real user data in indexes, and cache behaviour. Use shadow traffic or canaries to cover what staging can't.

PreviousWhat should be versioned in an LLM application?

This is what real progress feels like.

How do dev, staging and production environments work for LLM applications? What's tricky about them?

Practices

Related