Maintenance and drift after you ship
Maintenance is the work after you ship: watching for drift, cost and new model versions, re-running the evaluation suite and rebuilding the index when the ground shifts, and re-deploying when a change earns it.
Last updated: 15 Oct, 2026 · LLMOps
The Project: the production assistant lesson assembled an app you can run, track and evaluate. Shipping it is not the end. The LLMOps lifecycle video names monitoring and maintenance as the stage after deployment, and sets cost alongside it: an LLM app makes paid API calls, so controlling that cost is part of keeping it healthy. This lesson is the maintenance stage: what to watch, what to re-run, and when to re-deploy.
What to watch
- Drift. The inputs change even when the app does not. New products, new phrasings and new questions arrive, and an app that answered well at launch starts missing, because the world moved and the index and prompts did not.
- Cost. Every model call is paid, so a rise in traffic or a longer prompt raises the bill. Watching cost per request catches a change that quietly made each answer more expensive.
- New model versions. Providers release and retire models. A newer model can be cheaper or better, and a retired one can stop your app, so a provider's changes are a signal to act on, not to ignore.
What to re-run, and when to re-deploy
Watching tells you something moved; re-running tells you whether it matters. The evaluation suite from earlier in the course is the first thing to re-run, on current inputs rather than the launch set, so a drop in score is seen as a number. When the data has moved, the RAG index is rebuilt from the current documents so retrieval sees today's facts. A re-deploy follows a change that earned it: a new model that scored better on the suite, a rebuilt index, or a prompt fix, each versioned so the new build is labelled and the old one can be restored.
From a signal to a re-deploy
| Signal | What you re-run | What a re-deploy carries |
|---|---|---|
| Answers drifting | The evaluation suite on current inputs | A new prompt or a rebuilt index |
| Rising cost | A cost check per request | A cheaper model or a shorter prompt |
| A new model version | The suite against the new model | The new model, if it scored better |
| Stale documents | Retrieval over a rebuilt index | The rebuilt index |
Where maintenance matters
- A live assistant, where scores on current traffic, not the launch set, tell you whether it still answers well.
- A cost budget, where a watched cost per request keeps a traffic spike or a prompt change from a surprise bill.
- A provider change, where a retired model is caught before it breaks the app and a better one is adopted on the evidence of the suite.
What this course left out
The lifecycle is wide, and this course taught the spine of it. These are the families it names but does not teach, with where to go next for each.
| Topic | What it is for | Where to read |
|---|---|---|
| Serving open models yourself | Running an open model on your own GPUs with the batching that makes it affordable | The vLLM documentation |
| Kubernetes and autoscaling | Running many replicas, load testing them and scaling on demand | Deploying on Amazon EKS |
| Infrastructure as code | Declaring the cluster, the database and the secrets in reviewed files | The Terraform registry docs |
| Scheduled orchestration | Re-indexing, batch evaluation and jobs that survive a restart | The Airflow documentation |
| Judge-model evaluation | Scoring answers with a model instead of word overlap | DeepEval overview and RAGAS overview |
| Guardrails and governance | Blocking bad input and output, and gating an agent's tool calls | NeMo Guardrails overview |
| A model gateway | One endpoint in front of every provider, with routing, fallbacks and caching | LiteLLM overview |
| Fine-tuning for real | Running the training job this course only shows | Fine-tuning LLMs |
Related
- Previous: Project: the production assistant
- Next: Swapping the stand-ins for the real thing
- See also: The signals to watch
- See also: Evaluation in the lifecycle
- List three inputs the shop assistant might see this month that its launch evaluation set would miss.
- Write the cost-per-request check you would watch and the threshold that would prompt a look.
- Decide, for a provider retiring your model, the order of steps from the new model to a re-deploy.
This is what real progress feels like.