Capstone idea: your own operated LLM system
You have studied one full production system; now build your own. BrandGuardian audits video ads against FTC rules. Your capstone keeps that shape, take an input, run it through retrieval and an LLM, return a structured result, operate the whole thing, but on a domain you choose. This is a brief, not a walkthrough: it sets the target and the bar, and you build it.
Pick a domain
BrandGuardian is one instance of a general shape: an input that needs checking or transforming against a body of knowledge, judged by a model, returned as something a system can act on. Pick your own version of that. A few that fit the shape without needing Azure:
- Resume screener. A job description plus a resume, checked against a rubric, returned as a fit score with reasons.
- Support triage. A customer message, matched against a help centre, returned with a category, a priority and a draft reply.
- Content moderation. A post, checked against community rules, returned with the rules it breaks and a severity.
- Contract review. A clause, checked against a policy, returned with the risk and the clause it conflicts with.
- Release notes auditor. A changelog, checked against a style guide, returned with what is missing.
The shape it must have
Whatever the domain, the system is the roadmap in one application, the same pieces BrandGuardian wires together:
| Piece | What yours does | Where the roadmap taught it |
|---|---|---|
| A knowledge base | Your rules or reference documents, chunked, embedded and retrievable | The RAG data pipeline stage |
| A typed state | One object the steps pass along, with the input, the findings and the verdict | The project's graph state |
| A workflow | Two or more nodes that run in order: prepare the input, then judge it | The LangGraph workflow stage |
| Structured output | A fixed shape per finding: a category, a severity, a description | The auditor node |
| A served API | One endpoint that takes the input and returns the verdict as JSON | Structuring the app, and the backend |
| Observability | A trace per request and a metric you watch: latency, cost, a quality score | The observability and tracking stages |
| An evaluation gate | A small golden set of inputs with known verdicts, run before you ship | The tracking and evaluation stage |
The bar
It is done when you can do for your system what the BrandGuardian lesson does for its 146-second run: take one input end to end and show its trace, its structured result, its latency and its evaluation score. If any of those is missing, the roadmap names the stage to go back to.
- One input produces a structured, grounded verdict, not free text.
- The run is traced, and you can open it and read what the model saw.
- A golden set of at least ten inputs runs as a gate and a bad change fails it.
- The cost and latency of one request are numbers you can quote.
- A short README and a runbook: how to run it, what the alerts mean, how to roll back.
Every expert started right here.