Python, APIs and LLM basics
You cannot judge what you do not understand. Before you evaluate an AI system you need to read and write Python, know how a call to a model provider works over HTTP, and understand how a language model behaves once tokens, prompting and cost come into it. Then you build the two kinds of system you will spend the rest of the roadmap testing: a plain service, and a minimal agent that decides which tools to call. Testing an agent only makes sense once you have built one.
Python for AI
APIs for AI
LLM Fundamentals
OpenAI Agents SDKGolden datasets and metrics
Evaluation begins with deciding what good means, in numbers, before you measure anything. This stage builds the two things every eval rests on: a golden dataset that reflects the questions real users actually ask, and metrics that score an answer, including a model acting as judge. Get these right and the rest of the roadmap has something solid to stand on.
DeepEval
RAGASContinuous evaluation and tracing
A one-off score proves nothing tomorrow. This stage turns evaluation into something that runs on every change and keeps working in production: the same tests in CI comparing prompts and models, and an observability platform that traces every live run and links a result back to the exact prompt, model and inputs that produced it.
Promptfoo
LangfuseGuardrails and safety
Evaluation tells you whether a system is good. Guardrails decide what it is allowed to do at all. This stage covers the layers that keep a system safe in production: checks on every reply, a person in the loop before anything irreversible, one place that allows or denies each tool call with an audit trail, a sandbox for any code the model writes, and a clear map of where user data travels and what must never reach a prompt, a log or an eval set.
LangGraph
Guardrails AI
Agent Governance Toolkit
E2BRed teaming and adversarial testing
Guardrails only count once someone has seriously tried to get past them. This final stage is where you attack the system the way a real adversary would: run a catalogue of known jailbreaks and injections, then write attacks specific to your own domain, and turn what you find into a report a security review accepts. The findings are scoped against the standards teams actually use, the OWASP Top 10 for LLM Applications and MITRE ATLAS.
Garak
PyRITTrusted to ship
6 stages, 17 frameworks and 5 projects. Finish them and you will have shipped the kind of systems these teams hire for, with the lessons to back every one.
Browse all frameworksQuestions about this roadmap
What does an AI evaluation and safety engineer do?
They decide whether an AI feature is accurate, reliable and safe enough to ship, and they build the systems that keep it that way. The work runs from writing golden datasets and metrics, through scoring answers with a model as judge and wiring evaluation into CI, to guardrails on what the system may do and red teaming that attacks it before anyone else does.
Can I start this roadmap as a beginner?
Yes. The first stage teaches Python from your first line of code, then APIs and how language models work, and builds a small system for you to evaluate. Every later stage assumes only the stages before it.
Do I need an API key or a paid model account?
No. The courses use small stand-in models you write yourself, or a small open model that runs on your own computer, so no lesson needs an API key or a paid account. Most courses end with a lesson showing the one change that swaps in a real hosted model.
How is this different from the AI Forward Deployed Engineer roadmap?
That roadmap is about building an AI system and shipping it to a customer. This one is about proving a system is good and safe: it goes deep on evaluation, tracing, guardrails and red teaming, which the forward deployed roadmap only samples in a couple of stages. Different job, different day.
Is this the same as AI red teaming?
Red teaming is one part of it, the last stage. The role also owns evaluation: writing the datasets and metrics that say whether a system works, scoring it, and gating releases. Many teams put both under one person, because a red-team finding is only useful if you can measure that the fix worked.
Do I have to learn every framework on the roadmap?
No. Where two frameworks do the same job, a stop offers both and finishing either one completes it. Knowing one evaluation framework, one guardrails library and one red-team tool well matters more than knowing all of them a little.