Production generative AI
Production generative AI is the work of keeping an LLM application correct, fast, affordable and safe after it works on a laptop: measuring it, protecting it and changing it without breaking it.
Last updated: 08 Oct, 2026 · LangChain 1.4
A demo answers the handful of questions its author chose. A production app answers thousands that nobody chose, from users who type anything, while prices, models and providers change underneath it. This lesson is the map of what to watch, what to do about it, and which lesson holds the working version.
The loop after the first working version
Shipping is not the end of the build. Each stage below feeds the next, and the last one feeds the first: what the traces and the complaints show becomes new test cases for the next round.
| Concern | The question to ask | First step | Lesson |
|---|---|---|---|
| Quality | Is it still right after this change? | Run test cases with a judge | Evaluating LLM apps, Evaluating RAG |
| Safety | What can a user make it say or do? | Check input and output in code | Guardrails |
| Reliability | What happens when the provider is down? | Fallbacks, retries, a timeout | LLM gateways |
| Cost | What does one request cost, and ten thousand? | Count tokens, then cache and route | LLM gateways |
| Latency | How long until the user sees text? | Stream the reply | Streaming and batch |
| Control | Which actions need a person to approve? | Interrupt before risky tools | Human in the loop |
| Visibility | What did the app do for this user? | Trace every call | Observability tools, below |
Measuring tokens, time and cost for one call
Cost and latency start from numbers a single call already reports. The reply from a model comes with token counts, and a streamed reply shows when the first text arrived. Both come from the same loop.
start, first, final = time.perf_counter(), None, None
for chunk in model.stream("Explain what a vector database is in two sentences."):
if first is None and chunk.text:
first = time.perf_counter() - start # time to first visible text
final = chunk if final is None else final + chunk # chunks add up to the whole message
total = time.perf_counter() - start # time to the last token
usage = final.usage_metadata # {"input_tokens": ..., "output_tokens": ...}The cost of a call is its tokens times the provider's price, with input and output priced separately:
import time
from langchain.chat_models import init_chat_model
model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
PRICE_IN, PRICE_OUT = 0.15, 0.60 # dollars per million tokens (Groq's list price for this model)
start, first, final = time.perf_counter(), None, None
for chunk in model.stream("Explain what a vector database is in two sentences."):
if first is None and chunk.text:
first = time.perf_counter() - start # the first visible text has arrived
final = chunk if final is None else final + chunk
total = time.perf_counter() - start
usage = final.usage_metadata
cost = usage["input_tokens"] * PRICE_IN / 1e6 + usage["output_tokens"] * PRICE_OUT / 1e6
print(f"first text after {first:.2f} s, finished after {total:.2f} s")
print("tokens in/out:", usage["input_tokens"], usage["output_tokens"])
print("thinking tokens:", usage["output_token_details"]["reasoning"])
print(f"cost of this call: ${cost:.7f}")
print(f"cost of 100,000 such calls: ${cost * 100_000:.2f}")first text after 0.63 s, finished after 0.80 s tokens in/out: 81 104 thinking tokens: 20 cost of this call: $0.0000745 cost of 100,000 such calls: $7.45
Reading the measurement
- First text after 0.63 s, finished after 0.80 s. Streaming does not make the model faster. It lets the reader see text from 0.63 s on instead of waiting for 0.80 s, a gap that grows with the length of the reply.
- 81 tokens in, 104 out, 20 of them thinking. gpt-oss-120b is a reasoning model: its thinking tokens come before the visible answer, carry no text, and are billed as output. They are part of the wait before the first text appears.
- $0.0000745 for the call, $7.45 for 100,000 like it. The second number is a planning figure for one fixed prompt and reply length. Real prompts and replies vary, so measure the average over real traffic before budgeting.
Controlling cost
- Send fewer input tokens. A shorter prompt and fewer retrieved chunks cost less on every request.
- Cap the output. A model that writes a page when a sentence would do bills for the page. A reasoning model also bills its thinking tokens as output; the call in LLM gateways shows how large that share can be.
- Route by difficulty. A small model answers the easy requests and a larger one the hard ones. A gateway does this with aliases.
- Cache repeats. An identical request answered from storage costs no model call.
Controlling latency
- Time to first token matters most for chat. Streaming shows text as it is written, which is the pattern in Streaming and batch.
- Smaller models answer sooner. So do shorter prompts and shorter replies.
- Independent calls can run together. Batch them instead of waiting for each one in turn.
- Retrieval and tools add time. Each tool call in an agent is another round trip.
Handling failures and retired models
Providers rate-limit, time out and go down. Set a timeout on every call, retry the errors that clear on their own, and name a fallback model for the rest, which is what LLM gateways builds. Models are also retired on a schedule: models that older notebooks and videos use, such as llama-3.3-70b-versatile and qwen3-32b, are on Groq's deprecations page with their replacements. Keep model ids in one configuration place, pin the exact id, and re-run the test cases on the replacement before the old one is switched off.
Keeping the quality up
Run the test cases from Evaluating LLM apps before each release, and fail the release when a score drops. Once the app is live, score a sample of real conversations the same way. Retrieval-based apps add the metrics in Evaluating RAG.
Keeping the app safe
- Check what goes in and what comes out. Guardrails covers PII, blocked topics and checks that run in code.
- Keep keys out of code. Read them from environment variables, never commit them, and use one key per app so a leak can be revoked alone.
- Give tools the least access they need. An agent that can only read cannot delete, and risky actions can wait for approval (Human in the loop).
- Treat retrieved text and user input as untrusted. Either can carry instructions meant for the model.
Seeing what the app did
A trace records one request step by step: the prompt, each model and tool call, the tokens, the time and the cost. Without it, a wrong answer cannot be explained and a slow one cannot be located. Log what is needed to debug and no more, because prompts often hold personal data. Tools that collect traces include LangSmith, Langfuse and AgentOps, and a gateway logs every call that passes through it.
Shipping and changing the app
Put the app behind an API (APIs for AI builds one with FastAPI). Treat the prompt, the model id and the retrieval settings as a versioned bundle that changes together, and release a change to a small share of traffic before all of it, so a regression reaches few users and a rollback is one switch.
A prototype vs a production app
| Prototype | Production | |
|---|---|---|
| Test cases | A few questions tried by hand | A fixed set scored on every change |
| Model | Whatever worked first | A pinned id with a fallback |
| Failures | Restart and try again | Timeouts, retries and fallbacks |
| Cost | Unnoticed | Tracked per request |
| Visibility | Print statements | Traces and logs |
| Safety | Trusts the user | Checks on input, output and tools |
Where you use this map
- Before a launch. Walk the seven concerns in the table above and note which ones have no answer yet.
- After an incident. Find the concern the failure belongs to and the step that would have caught it.
- Choosing a tool. Each tool named below covers one row; the map shows which row is still open.
Related
- Previous: LLM gateways
- See also: Evaluating LLM apps and Guardrails
- Go deeper, tools: Langfuse, LiteLLM, DeepEval, RAGAS, Guardrails AI and AgentOps
- Go deeper, interview sets: LLMOps & Production Interview Questions, Evals & Guardrails Interview Questions, RAG Interview Questions and AI System Design Interview Questions
- Change the prompt to ask for a five-paragraph explanation. Compare the first-text time, the total time and the cost with this run.
- Change
PRICE_INandPRICE_OUTto the prices of another model and read how the cost of 100,000 calls moves. - Add
temperature=0to a second model line for gpt-oss-20b and run both: compare tokens, time and cost.
You understood something today that you didn't yesterday.