CI/CD for an LLM app

CI/CD for an LLM app is a pipeline that runs on every push: it lints the code, runs the tests, runs an evaluation set as a gate, builds the container and deploys, so a change ships only when the quality check passes.

Last updated: 15 Oct, 2026 · LLMOps

The The Azure deployment from the project lesson was the last of the where-to-deploy choices. This lesson is how a deploy happens without a person running commands by hand. The pattern is the one used across the MLOps work, carried into the Detailed LLMOps Project Lifecycle video's deployment stage: the steps that build and ship are automated, and the step that makes an LLM pipeline different is an evaluation gate between the tests and the deploy. It is shown as a GitHub Actions workflow, not run, because it runs on a push to a repository.

A workflow that gates the deploy on an eval

The workflow triggers on a push to main and runs one job of ordered steps. The lint and the unit tests are the usual CI; the step that matters for an LLM app is the evaluation run, placed before the build so a drop in answer quality fails the job and the deploy steps never run.

name: deploy
on:
  push:
    branches: [main]

jobs:
  ship:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install -r requirements.txt
      - run: ruff check .            # lint the code
      - run: pytest tests/eval       # the eval gate: fails the job if a score drops
      - run: docker build -t shop-assistant .
      - run: ./deploy.sh             # deploy, reached only when the eval passed

Why the eval is a gate, not a report

A unit test checks that the code runs; it cannot tell you the model's answers got worse after a prompt edit or a model swap. The evaluation set can, and making it a step that fails the job turns it from a report someone might read into a gate a bad change cannot pass. A cheaper model or a trimmed prompt from the cost lesson reaches production only after it clears this bar.

• Lint and unit tests catch a broken build and a code error, the same as any app's CI.
• The evaluation gate runs your scored examples and fails the job if the score falls below a threshold, so a quality regression blocks the deploy.
• Build and deploy run last, and only when every step before them passed, so nothing ships that did not clear the gate.

Code CI against an LLM app's CI

Where a CI/CD gate helps

• A prompt or model change, where the eval gate catches a quality drop that no unit test would see.
• A team shipping often, where automating build and deploy removes the manual step a person forgets.
• A regression guard, where the scored set grows with every bug found, so the same bug cannot ship twice.

Related

• Previous: The Azure deployment from the project
• Next: Versioning model, data and code
• The evaluation tool behind the gate: Promptfoo overview
• Running graded code safely in CI: E2B overview