Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Production generative AI

Production generative AI is the work of keeping an LLM application correct, fast, affordable and safe after it works on a laptop: measuring it, protecting it and changing it without breaking it.

Last updated: 08 Oct, 2026 · LangChain 1.4

A demo answers the handful of questions its author chose. A production app answers thousands that nobody chose, from users who type anything, while prices, models and providers change underneath it. This lesson is the map of what to watch, what to do about it, and which lesson holds the working version.

The loop after the first working version

Shipping is not the end of the build. Each stage below feeds the next, and the last one feeds the first: what the traces and the complaints show becomes new test cases for the next round.

Five boxes in a row, Build, Evaluate, Guard, Deploy and Observe, each with an arrow to the next. A dashed arrow runs from Observe back under the row to Evaluate: what the traces and the complaints show becomes new test cases.
ConcernThe question to askFirst stepLesson
QualityIs it still right after this change?Run test cases with a judgeEvaluating LLM apps, Evaluating RAG
SafetyWhat can a user make it say or do?Check input and output in codeGuardrails
ReliabilityWhat happens when the provider is down?Fallbacks, retries, a timeoutLLM gateways
CostWhat does one request cost, and ten thousand?Count tokens, then cache and routeLLM gateways
LatencyHow long until the user sees text?Stream the replyStreaming and batch
ControlWhich actions need a person to approve?Interrupt before risky toolsHuman in the loop
VisibilityWhat did the app do for this user?Trace every callObservability tools, below

Measuring tokens, time and cost for one call

Cost and latency start from numbers a single call already reports. The reply from a model comes with token counts, and a streamed reply shows when the first text arrived. Both come from the same loop.

python
start, first, final = time.perf_counter(), None, None
for chunk in model.stream("Explain what a vector database is in two sentences."):
    if first is None and chunk.text:
        first = time.perf_counter() - start        # time to first visible text
    final = chunk if final is None else final + chunk   # chunks add up to the whole message
total = time.perf_counter() - start               # time to the last token
usage = final.usage_metadata                      # {"input_tokens": ..., "output_tokens": ...}

The cost of a call is its tokens times the provider's price, with input and output priced separately:

The cost of one call; the prices are dollars per token
ExampleAPI keyLive run on Groq, gpt-oss-120b; the prices are Groq's list prices at the time of writing
import time
from langchain.chat_models import init_chat_model

model = init_chat_model("groq:openai/gpt-oss-120b", temperature=0)
PRICE_IN, PRICE_OUT = 0.15, 0.60          # dollars per million tokens (Groq's list price for this model)

start, first, final = time.perf_counter(), None, None
for chunk in model.stream("Explain what a vector database is in two sentences."):
    if first is None and chunk.text:
        first = time.perf_counter() - start        # the first visible text has arrived
    final = chunk if final is None else final + chunk
total = time.perf_counter() - start

usage = final.usage_metadata
cost = usage["input_tokens"] * PRICE_IN / 1e6 + usage["output_tokens"] * PRICE_OUT / 1e6
print(f"first text after {first:.2f} s, finished after {total:.2f} s")
print("tokens in/out:", usage["input_tokens"], usage["output_tokens"])
print("thinking tokens:", usage["output_token_details"]["reasoning"])
print(f"cost of this call: ${cost:.7f}")
print(f"cost of 100,000 such calls: ${cost * 100_000:.2f}")

Reading the measurement

  • First text after 0.63 s, finished after 0.80 s. Streaming does not make the model faster. It lets the reader see text from 0.63 s on instead of waiting for 0.80 s, a gap that grows with the length of the reply.
  • 81 tokens in, 104 out, 20 of them thinking. gpt-oss-120b is a reasoning model: its thinking tokens come before the visible answer, carry no text, and are billed as output. They are part of the wait before the first text appears.
  • $0.0000745 for the call, $7.45 for 100,000 like it. The second number is a planning figure for one fixed prompt and reply length. Real prompts and replies vary, so measure the average over real traffic before budgeting.

Controlling cost

  • Send fewer input tokens. A shorter prompt and fewer retrieved chunks cost less on every request.
  • Cap the output. A model that writes a page when a sentence would do bills for the page. A reasoning model also bills its thinking tokens as output; the call in LLM gateways shows how large that share can be.
  • Route by difficulty. A small model answers the easy requests and a larger one the hard ones. A gateway does this with aliases.
  • Cache repeats. An identical request answered from storage costs no model call.

Controlling latency

  • Time to first token matters most for chat. Streaming shows text as it is written, which is the pattern in Streaming and batch.
  • Smaller models answer sooner. So do shorter prompts and shorter replies.
  • Independent calls can run together. Batch them instead of waiting for each one in turn.
  • Retrieval and tools add time. Each tool call in an agent is another round trip.

Handling failures and retired models

Providers rate-limit, time out and go down. Set a timeout on every call, retry the errors that clear on their own, and name a fallback model for the rest, which is what LLM gateways builds. Models are also retired on a schedule: models that older notebooks and videos use, such as llama-3.3-70b-versatile and qwen3-32b, are on Groq's deprecations page with their replacements. Keep model ids in one configuration place, pin the exact id, and re-run the test cases on the replacement before the old one is switched off.

Keeping the quality up

Run the test cases from Evaluating LLM apps before each release, and fail the release when a score drops. Once the app is live, score a sample of real conversations the same way. Retrieval-based apps add the metrics in Evaluating RAG.

Keeping the app safe

  • Check what goes in and what comes out. Guardrails covers PII, blocked topics and checks that run in code.
  • Keep keys out of code. Read them from environment variables, never commit them, and use one key per app so a leak can be revoked alone.
  • Give tools the least access they need. An agent that can only read cannot delete, and risky actions can wait for approval (Human in the loop).
  • Treat retrieved text and user input as untrusted. Either can carry instructions meant for the model.

Seeing what the app did

A trace records one request step by step: the prompt, each model and tool call, the tokens, the time and the cost. Without it, a wrong answer cannot be explained and a slow one cannot be located. Log what is needed to debug and no more, because prompts often hold personal data. Tools that collect traces include LangSmith, Langfuse and AgentOps, and a gateway logs every call that passes through it.

Shipping and changing the app

Put the app behind an API (APIs for AI builds one with FastAPI). Treat the prompt, the model id and the retrieval settings as a versioned bundle that changes together, and release a change to a small share of traffic before all of it, so a regression reaches few users and a rollback is one switch.

A prototype vs a production app

PrototypeProduction
Test casesA few questions tried by handA fixed set scored on every change
ModelWhatever worked firstA pinned id with a fallback
FailuresRestart and try againTimeouts, retries and fallbacks
CostUnnoticedTracked per request
VisibilityPrint statementsTraces and logs
SafetyTrusts the userChecks on input, output and tools

Where you use this map

  • Before a launch. Walk the seven concerns in the table above and note which ones have no answer yet.
  • After an incident. Find the concern the failure belongs to and the step that would have caught it.
  • Choosing a tool. Each tool named below covers one row; the map shows which row is still open.
Watch out. Prices, rate limits and model names change. The cost lines above use one provider's list price at the time of writing; read the live pricing page before planning a budget, and re-measure after any model change.
Try it yourself
  • Change the prompt to ask for a five-paragraph explanation. Compare the first-text time, the total time and the cost with this run.
  • Change PRICE_IN and PRICE_OUT to the prices of another model and read how the cost of 100,000 calls moves.
  • Add temperature=0 to a second model line for gpt-oss-20b and run both: compare tokens, time and cost.
PreviousLLM gateways

You understood something today that you didn't yesterday.