What's different about containerising LLM applications and model servers?

30-second answerSay your answer out loud first, then reveal.

Application containers (orchestration, RAG, API) are normal and lightweight. Model server containers are different: they need GPU drivers and CUDA compatibility, large model weights (tens to hundreds of GB) that shouldn't be baked into images, careful startup (weight loading can take minutes), health and readiness checks that reflect model readiness, and resource requests for GPUs. Separate the model weights from the image, cache them on local NVMe or shared volumes, and pin inference engine and driver versions.

Practices

Concern	Approach
Image size	Slim base images; inference engine + dependencies only; weights downloaded or mounted at startup
Weights	Model registry / object storage → local NVMe cache or a pre-populated volume; checksum verification
Startup time	Readiness probe only after the model is loaded and warmed up; pre-pulled images on GPU nodes; fast weight loading (safetensors, parallel download)
GPU compatibility	Match CUDA / driver versions; use GPU operator-managed nodes; pin engine versions
Resources	Request whole GPUs (or MIG slices); set shared memory size for tensor parallelism
Health	Liveness (process alive) vs readiness (model loaded, can serve); a test generation in the readiness check
Security	Non-root, minimal packages, signed images, no secrets baked in

Why it matters: slow startup affects autoscaling and recovery. A pod that takes 8 minutes to become ready can't absorb a sudden spike.

PreviousHow should API keys and secrets be managed in LLM applications?

Every expert started right here.

What's different about containerising LLM applications and model servers?

Practices

Related