How do you run GPU inference workloads on Kubernetes?

30-second answerSay your answer out loud first, then reveal.

Use dedicated GPU node pools (with the NVIDIA GPU Operator or managed equivalents for drivers and device plugins), schedule model-server pods with GPU resource requests, node selectors and taints and tolerations, and use an inference engine (vLLM, SGLang, TGI, TensorRT-LLM / Triton) behind a service and load balancer. Autoscale on inference-specific metrics (queue depth, KV-cache usage, latency) using tools like KEDA or custom HPA metrics, and handle slow cold starts with warm capacity and pre-pulled images and weights.

GPU inference on Kubernetes: an ingress feeds a load-aware model service that routes to vLLM pods loading weights from object storage into a node NVMe cache; pod metrics (queue depth, KV-cache %, TTFT, tokens/s) drive a KEDA or custom HPA autoscaler that scales the pods and triggers a node autoscaler for the GPU node pool.

Key considerations

Node pools: separate pools per GPU type (e.g. 80 GB vs 24 GB cards); taints so only GPU workloads land there.
Pod sizing: one model replica may span multiple GPUs (tensor parallelism); request nvidia.com/gpu: N; set /dev/shm size.
GPU sharing: MIG partitions or time-slicing for small models (with caution for latency-sensitive workloads).
Routing: least-outstanding-requests or KV-cache-aware routing beats round-robin for LLMs; prefix-cache affinity improves hit rates.
Cold starts: node provisioning + image pull + weight loading can take 5–15 minutes, so keep minimum replicas, use predictive scaling, pre-pull images and cache weights.
Disruption: PodDisruptionBudgets; graceful shutdown that drains in-flight streams.
Observability: DCGM exporter for GPU metrics and inference engine metrics in Prometheus/Grafana.

PreviousHow do you design rollback for an LLM application? What exactly do you roll back?

This is what real progress feels like.

How do you run GPU inference workloads on Kubernetes?

Key considerations

Related