Deployment choices, and why
A deployment choice for an LLM app is the decision of where the model runs and how your code reaches it: a managed provider API, or a model you host on your own cloud, served behind an endpoint that something has to scale and pay for.
Last updated: 15 Oct, 2026 · LLMOps
Part 4 turned the shop assistant into a project you can run and test, and the monitoring part watched it once it was live. Before any of that is live, you pick where it runs. The Detailed LLMOps Project Lifecycle video puts deployment near the end of the lifecycle and sums the stage up as infrastructure setup, API, endpoint, scaling strategies on AWS or Azure or whatever cloud you are using, with one question he calls important: how do we optimize the API call cost. This lesson maps the choices that stage opens.
The first fork: a managed API or your own model
The largest decision is whether the model is someone else's service or your own. A managed API, such as a provider's hosted model, is a key and an HTTP call: nothing to host, the provider scales it, and you pay per token. Your own model means you take an open-source model and run it on hardware you rent, so the weights and the traffic never leave your infrastructure.
The lifecycle video names the case for the second path: a company whose requirement is we need to deploy in their own cloud the fine-tuned model. When data privacy or a tuned model is the point, a managed API will not do, and the app is served from a model the company hosts itself.
How a hosted model is served
Once you host the model, the code reaches it through one of three shapes, from least to most operational work.
- Serverless runs your code only when a request arrives and scales to zero between requests. It suits bursty or low traffic and a small app, but a cold start adds latency and a long-running model load fits it poorly.
- A container packages the app and its dependencies into one image that runs the same on any host. It is the common middle ground: always-on, portable, and the unit a cloud service or an orchestrator schedules.
- A model endpoint is a managed service, such as a SageMaker endpoint, that loads the model on a GPU instance and exposes it over HTTP. You hand it the model and the instance type; the platform runs and scales the serving for you.
The choices side by side
| Managed provider API | Your own hosted model | |
|---|---|---|
| You run | Nothing; the provider hosts it | A container or a model endpoint you own |
| Data leaves your cloud | Yes, to the provider | No, which is the privacy case |
| You pay for | Tokens per call | The instance or GPU by the hour |
| Scaling is | The provider's problem | Yours to set up |
| Fits | Most apps, fastest to ship | Privacy rules or a fine-tuned model |
Where each choice fits
- A managed provider API, where speed to ship matters and the data may leave your cloud, which covers most apps.
- A container on your cloud, where you host an open-source model but want it portable and always on.
- A model endpoint on a GPU instance, where a company's rule is that a tuned model and its traffic stay inside their own cloud.
Related
- Previous: The signals to watch
- Next: Packaging the app in a container
- The app this part deploys: The app entry point and its API
- For an app you run, write down which of the two paths it is on and the one requirement that decided it.
- List what you would have to operate yourself to move a provider-API app onto your own GPU instance.
- Read the deployment stage of the Detailed LLMOps Project Lifecycle video and note the four words the video uses for it: infrastructure, API, endpoint, scaling.
This is what real progress feels like.