Why a gate

The gate in Python

Rules in a file

Rules that read the situation

The audit trail

Zero-trust identity

MCP governance

The command line

Running it safely

Project

Getting started

LLM security and guardrails

NeMo Guardrails

LLM evaluation

RAG evaluation metrics

Short-term agent memory

Long-term agent memory

Managing agent memory

AgentOps: the production stack

Deploying and scaling

LLM as a judge

Evaluating RAG

Datasets and runs

Agents and conversations

Shipping the suite

Generative AI foundations

Building with LLMs

AI agents

Loop engineering

Harness engineering

Quantization and fine-tuning

Production

Starting out

History

Branches

GitHub

Undoing things

A repository a team can read

Your first trace

Model calls

Organising traces

What leaves your process

Prompts

Scores and experiments

The traced desk, end to end

Appendix

Why a gateway

The SDK, call by call

Tools, shapes and a model of your own

The Router

Paying less

The gateway

Running it for a company

The gateway you would run

Operating the gateway

How a model writes

Controlling the output

In production

What LLMOps is

Planning and the pipeline

Data and model selection

Structuring an LLM app for production

Tracking and evaluation

Monitoring and observability

Deployment

The end-to-end project

The project: a compliance auditor

Before the guardrails

The runtime and a model

Colang: teaching it the conversation

Input rails

Actions and a rail of your own

Output rails

Dialog and execution rails

Running it for real

The guarded IT assistant

First lines of Python

Many values

Functions

Files, JSON and errors

Classes and objects

Advanced Python

Types and validation

Waiting on models

A real project

Evaluating a RAG app

Metrics without a judge

The five RAG metrics

Your own metrics

Running the evaluation

Shipping the evaluation

Agent Governance Toolkit

DeepEval

LiteLLM

LLM Fundamentals

Python for AI

RAGAS

Langfuse

Generative AI and Agentic AI

AI Security

Git and GitHub

LLMOps

Platform engineer

Errors you can catch, classes and dataclasses, type hints, Pydantic, and async calls.

Python a service is written in

A virtual environment per project, pytest, and logging instead of print.

Environments, tests and logging

Commits, history, branches, conflicts, remotes and pull requests.

Git and GitHub: the first step

Amend, revert, reset, stash, repository structure, and large files.

Undoing things, and what must never be committed

What a model call really costs, and the knobs that change it.

Tokens, context, cost and latency

Python, Git and how a model behaves

What changes from MLOps, why an LLM app needs ops at all, the ten lifecycle stages, the team, and the four use cases.

The LLMOps lifecycle

What an LLM app is made of: prompts, messages, structured output, streaming and memory.

The workload: a generative AI app end to end

What LLMOps is, and the lifecycle

What the app must do, the pipeline that does it, and whether that means RAG, fine-tuning or both.

Requirements, pipeline shape, and the build decision

Ingest, parse, split, chunk, embed and index for RAG; the dataset shape for fine-tuning.

The data pipeline for each build

Providers, model cards, size and cost, and what changes once the model can call tools.

Choosing a model, and giving it tools

Loading, chunking, embeddings, a pipeline, better retrieval, conversational and agentic RAG, and evaluating it.

The RAG build, in full

Quantization, LoRA and QLoRA, and fine-tuning on custom data.

The fine-tuning build, and quantization

Plan the build: pipeline, data and model

A project a stranger can navigate, settings out of the code, logs you can search, and errors that say where they came from.

Layout, config, logging and exceptions

Prompts in one place, one API in front of the app, and a project that installs.

A prompt library, an entry point and packaging

Document Q&A API: answer questions about a PDF

A small service that answers questions about an uploaded PDF, built the way the last two stops describe.

The app you will operate

Structure the app for production

completion(), the model string, aliases, streaming, cost, one exception set, and tool calling.

One interface for every provider

Model groups, routing strategies, retries, fallbacks, cooldowns, the cheap model first, and three kinds of cache.

Routing, retries, fallbacks and caching

The proxy, the OpenAI client against it, gateway keys, a container, virtual keys and the admin UI.

The gateway a team runs

One gateway in front of every model

Runs, parameters and metrics, logging a GenAI evaluation, what to evaluate, and a remote tracking server.

Experiment tracking with MLflow

Golden datasets: knowing an AI system works

The small set of cases that must keep working, and how to build one that is worth trusting.

The golden dataset

Goldens, exact match, context precision and recall, judges, faithfulness, relevancy and correctness.

Evaluating retrieval properly

Safety metrics, a test run, evals in CI, integrations, and a full evaluation project.

Evaluation as a test that can fail

Track experiments, prove quality

What goes wrong in production, tracing a run with LangSmith, the studios, and the signals that matter.

Why you monitor, and what to watch

Traces and observations, users, sessions, environments, masking, sampling and batching.

Tracing with Langfuse

Prompts managed outside the code, versions and labels, scores, feedback, datasets, experiments and a gate.

Prompts, scores and a regression gate

Observability across every call, health checks, Prometheus metrics and alerting.

Metrics and alerts at the gateway

Observability, signals and cost

Where to deploy and why, packaging the app in a container, serving and scaling the API, and controlling what calls cost.

Containers, an API, scale and cost

An open-source LLM on SageMaker, and the Azure deployment from the end-to-end project.

Deploying on a cloud

A pipeline that tests, gates and deploys on a push, and what you version besides code.

CI/CD, and versioning everything

An agentic RAG API with FastAPI and LangGraph, deployed on EKS, load tested with Locust, and autoscaled.

Kubernetes, load testing and autoscaling

A high-throughput inference server, with the batching that makes self-hosting affordable.

Serving open models yourself

The cluster, the database and the secrets as a reviewed file rather than a console session.

Infrastructure as code

Containers, cloud and CI/CD

The OWASP LLM top ten, guardrails, prompt injection and jailbreaks, frameworks, gateways, and measuring a rail.

The actual attack surface

Rails on the way in and the way out, refusals you can explain, custom actions and rewritten answers.

Input and output rails

One choke point, allow-lists and default deny, argument checks, policy rules in YAML and which rule wins.

A policy gate on every tool call

Where decisions are written, the flight recorder, querying it, and proving it was not edited.

An audit trail you can prove

Customer data and PII: what an AI system may see

What you may send to a provider, what you must not log, and what the retention rules mean in practice.

Customer data and PII

Guardrails, governance and data

Observability and orchestration once it is live, and what you re-check as the world moves.

Production operations, drift and maintenance

The runnable build that carries every stage, then the seam-by-seam swap to real services.

The keyless assistant, and swapping in the real thing

The jobs around the model: re-indexing, batch evaluation, retraining, and work that must survive a restart.

Scheduled and durable orchestration

Handover: leaving a system a customer's team can run

What someone else needs in order to take the pager from you.

Runbooks and handover

The full production system every stage of this roadmap has been building toward, studied node by node.

The project this roadmap builds: BrandGuardian

Capstone idea: your own operated LLM system

A brief, not a walkthrough. Take BrandGuardian's shape to a domain you choose, and operate it to the same bar.

Capstone: build your own

Keep it healthy, and hand it over

An AI platform or LLMOps engineer owns everything around the model: the repository the app lives in, the gateway every service calls through, the experiment that proved a prompt was better, the traces that explain last night's bad answer, the container and the pipeline that ship it, and the bill at the end of the month. This roadmap walks that lifecycle in order. It starts with Python and version control, because nothing later works without them, then follows the LLMOps course stage by stage, bringing in a full framework course wherever a stage needs a real tool. The roadmap is built around one real project, BrandGuardian, a multimodal compliance auditor that checks video ads against FTC rules with LangGraph, Azure AI Search and GPT-4o, observed with LangSmith and Application Insights. Every stage teaches a piece of what it is made of; you study the full build at the end, then take its shape to a domain of your own as the capstone.