How should prompts be managed in production?

Practices

1. Versioning: each prompt has an ID and a version; changes go through pull requests.
2. Templates: structured sections (role, instructions, examples, context slots) with typed variables. Validate the variables.
3. Separation of concerns: keep model parameters (temperature, max tokens), tool schemas and prompt text together as one "prompt config" unit.
4. Eval gates: run the regression suite before merging; compare against the baseline.
5. Rollout: feature flags, canary or A/B, fast rollback.
6. Traceability: log prompt version + model version per request.
7. Ownership: product, domain experts and engineers may all edit prompts. A registry UI with review workflows helps non-engineers contribute safely.
8. Caching-friendly structure: stable prefix first, dynamic content last (Q6).

Tools: git + config files, LangSmith/Langfuse/Braintrust prompt management, or an internal registry.

Common mistakes

• Prompts hard-coded in many places across the codebase with no tests.
• "Quick fixes" to prompts in production that silently break other cases.

Related

• Previous: Q11. What is a data flywheel for an AI product, and how do you design feedback collection?
• Next: Q13. When should you use classical ML instead of an LLM in a system?