Deploying stateless web microservices on Kubernetes is a well-established engineering practice. You package your application in a container, define resource requests and limits, attach an ingress controller, and configure the Horizontal Pod Autoscaler (HPA) to scale pods based on CPU and memory thresholds.
When you introduce AI inference engines and large language model (LLM) serving into a Kubernetes cluster, this traditional operational playbook breaks down completely.
Running a local model using ollama run on an engineer’s workstation requires almost no infrastructure consideration. In contrast, serving high-concurrency LLM inference in production presents severe architectural constraints:
- Modern inference engines require tens of gigabytes of dedicated High-Bandwidth Memory (HBM) on expensive GPU hardware.
- Model weight artifacts (often 30GB to 100GB+) cause severe cold-start delays, turning standard pod rescheduling into minutes of downtime.
- Traditional CPU-based autoscaling fails because GPU memory is allocated upfront for Key-Value (KV) caches, masking true workload saturation.
- An idle multi-GPU node (such as an NVIDIA H100 or A100 instance) can cost thousands of dollars per month, making over-provisioning financially unsustainable.
This guide provides a comprehensive infrastructure deep dive into running AI workloads on Kubernetes, covering GPU scheduling, inference architecture, production observability, and battle-tested cost optimization strategies.
Core Components of an AI Infrastructure Stack on Kubernetes
A production-grade AI serving platform requires tight integration across specialized hardware, kernel drivers, container runtimes, and orchestration controllers:
+-------------------------------------------------------------------------------+
| AI Inference Architecture on Kubernetes |
+-------------------------------------------------------------------------------+
+───────────────────────────────────────────────────────────────────────────+
| Client Application Layer |
+───────────────────────────────────────────────────────────────────────────+
│
│ OpenAI-Compatible HTTP / gRPC
▼
+───────────────────────────────────────────────────────────────────────────+
| API Gateway & Ingress Layer |
| (Envoy / Istio / KServe / Rate Limiting & Auth Tokens) |
+───────────────────────────────────────────────────────────────────────────+
│
│ High-Throughput Request Routing
▼
+───────────────────────────────────────────────────────────────────────────+
| Inference Serving Engine |
| (vLLM / NVIDIA Triton / TGI - Continuous Batching & PagedAttention) |
+───────────────────────────────────────────────────────────────────────────+
│
▼
+───────────────────────────────────────────────────────────────────────────+
| Kubernetes GPU Scheduling & Orchestration |
| (NVIDIA GPU Operator / Device Plugin / DRA / MIG / KEDA) |
+───────────────────────────────────────────────────────────────────────────+
│
┌──────────────┴──────────────┐
▼ ▼
[ High-Speed Local NVMe ] [ Distributed Object Store ]
(Model Weight Fast Cache) (S3 / GCS / Ceph / NFS)
│ │
└──────────────┬──────────────┘
▼
+───────────────────────────────────────────────────────────────────────────+
| Physical Compute Hardware |
| (NVIDIA H100 / A100 / L40S / AMD Instinct / NVLink & InfiniBand) |
+───────────────────────────────────────────────────────────────────────────+Essential Stack Layers:
- Model Weights & Cache Storage: Multi-gigabyte safetensors stored in object storage and synchronized to local host NVMe volumes to prevent network bottlenecks during pod restarts.
- Inference Serving Engines (vLLM, NVIDIA Triton, HuggingFace TGI): Optimized runtimes that replace standard web frameworks with C++/CUDA kernels implementing PagedAttention and Continuous Batching.
- GPU Drivers & Operator: The NVIDIA GPU Operator automates kernel driver installation, CUDA container runtimes, and DCGM monitoring daemonsets.
- Intelligent Ingress & Gateway: Ingress controllers (like KServe or Envoy) that route requests based on token queue depth rather than simple round-robin algorithms.
Why AI Workloads Differ from Traditional Microservices
Understanding the operational differences between standard applications and AI inference is essential for cluster stability:
| Operational Dimension | Traditional Web Microservices | AI & LLM Inference Workloads |
|---|---|---|
| Primary Bottleneck | CPU cycles, network I/O, database latency. | GPU High-Bandwidth Memory (HBM), PCIe/NVLink bandwidth. |
| Startup & Cold Starts | 200ms to 5 seconds. | 45 seconds to 5 minutes (downloading & loading weights into VRAM). |
| Autoscaling Triggers | CPU utilization percentage ($>70%$), Memory usage. | Request queue depth, KV-cache memory saturation, token throughput. |
| Resource Allocation | Fractional vCPU and dynamic memory allocations. | Discrete GPU devices (or rigid MIG hardware slices). |
| Failure Blast Radius | Low; pods reschedule and start immediately. | High; pod crashes trigger long model loading delays and GPU idle waste. |
| Hardware Costs | Predictable, low-to-moderate compute pricing. | High; individual GPU nodes cost $$2,000$ to $$8,000+$ monthly. |
Not Every AI Workload Needs a GPU
A common and expensive architectural mistake is assuming that every machine learning model requires an enterprise GPU node.
+-------------------------------------------------------------------------------+
| CPU vs GPU Workload Decision Matrix |
+-------------------------------------------------------------------------------+
RUN ON MODERN CPU NODES (Cost-Effective):
┌───────────────────────────────────────────────────────────────────────────┐
│ • Text embeddings generation (e.g., all-MiniLM-L6-v2, bge-small) │
│ • Classical ML (XGBoost, Random Forest, Scikit-Learn models) │
│ • Small quantized LLMs (e.g., 3B-7B models with INT4/INT8 via AVX-512/AMX)│
│ • Low-throughput, internal asynchronous batch jobs │
└───────────────────────────────────────────────────────────────────────────┘
RUN ON DEDICATED GPU NODES (Performance-Critical):
┌───────────────────────────────────────────────────────────────────────────┐
│ • Real-time streaming LLM inference (e.g., 14B, 32B, 70B parameters) │
│ • High-concurrency production API endpoints requiring p99 < 800ms TTFT │
│ • Computer vision models (Stable Diffusion, YOLO, OCR at high frame rates) │
│ • Model fine-tuning, LoRA training, and distributed training pipelines │
└───────────────────────────────────────────────────────────────────────────┘By offloading embeddings generation and lightweight classification tasks to modern CPU nodes (utilizing Intel AMX or AMD AVX-512 extensions), infrastructure teams can reduce GPU cluster costs by up to 60%.
Kubernetes Scheduling Primitives for GPU Workloads
To ensure AI pods land on the correct physical nodes without starving other workloads, Kubernetes relies on specific scheduling primitives:
1. Taints and Tolerations
GPU nodes should always be tainted to prevent non-AI web services from consuming memory on expensive hardware:
# Node Taint applied to GPU instances:
key: "nvidia.com/gpu"
value: "present"
effect: "NoSchedule"2. Resource Requests and Limits
Unlike CPU, standard Kubernetes GPU resources are discrete integers and cannot be oversubscribed. Requests and limits must be identical:
resources:
limits:
nvidia.com/gpu: 1
requests:
nvidia.com/gpu: 13. GPU Virtualization: MIG vs Time-Slicing
- NVIDIA Multi-Instance GPU (MIG): Physically partitions an A100 or H100 GPU into up to seven fully isolated hardware instances with dedicated compute cores and isolated memory bandwidth. Best for multi-tenant production security.
- Time-Slicing: Allows multiple pods to share a single GPU by multiplexing compute over time. While it improves density, it provides zero memory isolation—if one pod exceeds VRAM, all co-located pods crash with CUDA Out-of-Memory (OOM) errors.
Production Observability: Why Traditional Monitoring Is Blind
Standard monitoring tools (like basic cAdvisor or node_exporter) only report host CPU and RAM metrics. They cannot see GPU tensor core activity or VRAM allocations.
+-------------------------------------------------------------------------------+
| Essential Metrics for Production AI Clusters |
+-------------------------------------------------------------------------------+
1. DCGM Metrics (NVIDIA Data Center GPU Manager):
- `dcgm_gpu_utilization`: Percentage of GPU compute cores active.
- `dcgm_fb_used` / `dcgm_fb_free`: Framebuffer (VRAM) memory consumption.
- `dcgm_nvlink_bandwidth`: Inter-GPU communication throughput.
2. vLLM / Inference Engine Metrics:
- `vllm:num_requests_waiting`: Number of requests queued in memory.
- `vllm:gpu_cache_usage_factor`: Percentage of VRAM allocated to KV cache.
- `vllm:time_to_first_token_seconds`: Time-to-First-Token (TTFT) latency.
- `vllm:time_per_output_token_seconds`: Inter-token generation latency (TPOT).Autoscaling AI workloads must be driven by vllm:num_requests_waiting using KEDA (Kubernetes Event-driven Autoscaling). Scaling based on CPU load will fail to scale during queue surges because GPU workers maintain constant CPU poll loops.
How to Control AI Infrastructure Costs (FinOps)
Operating AI workloads without strict FinOps guardrails can quickly lead to massive budget overruns. Apply these eight cost-optimization strategies:
+-------------------------------------------------------------------------------+
| 8 Pillars of AI Infrastructure FinOps |
+-------------------------------------------------------------------------------+
1. Model Quantization ──► Use AWQ / FP8 to fit 70B models onto single GPUs.
2. Continuous Batching ──► Maximize token throughput per GPU dollar.
3. Autoscaling on Queue ──► Scale pods using KEDA based on waiting requests.
4. Scale-to-Zero (Staging)─► Shut down test inference endpoints during off-hours.
5. Spot/Preemptible GPUs ──► Save up to 70% on fault-tolerant batch workloads.
6. Right-Sizing Compute ──► Use NVIDIA L4 / L40S instead of H100 for small models.
7. Local NVMe Caching ──► Avoid redundant multi-gigabyte egress downloads.
8. Workload Attribution ──► Track cost per token per engineering team via Kubecost.Production Kubernetes Manifest: Deploying a vLLM Inference Service
Below is a production-ready Kubernetes deployment for serving an open-weights LLM using vLLM with proper resource limits, shared memory mounts, and readiness probes:
apiVersion: apps/v1
kind: Deployment
metadata:
name: vllm-llama-service
namespace: ai-inference
labels:
app: vllm-inference
spec:
replicas: 2
selector:
matchLabels:
app: vllm-inference
template:
metadata:
labels:
app: vllm-inference
spec:
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
containers:
- name: vllm-engine
image: vllm/vllm-openai:v0.6.2
args:
- "--model"
- "meta-llama/Llama-3.1-8B-Instruct"
- "--port"
- "8000"
- "--max-model-len"
- "8192"
- "--gpu-memory-utilization"
- "0.90"
env:
- name: HUGGING_FACE_HUB_TOKEN
valueFrom:
secretKeyRef:
name: hf-token-secret
key: token
ports:
- containerPort: 8000
name: http
resources:
limits:
nvidia.com/gpu: "1"
memory: "32Gi"
cpu: "8"
requests:
nvidia.com/gpu: "1"
memory: "16Gi"
cpu: "4"
volumeMounts:
- mountPath: /dev/shm
name: dshm
- mountPath: /root/.cache/huggingface
name: model-cache
readinessProbe:
httpGet:
path: /health
port: 8000
initialDelaySeconds: 45
periodSeconds: 10
volumes:
- name: dshm
emptyDir:
medium: Memory
sizeLimit: 8Gi
- name: model-cache
persistentVolumeClaim:
claimName: ai-model-nvme-pvcKey Configuration Directives:
/dev/shmVolume Mount: PyTorch and CUDA utilize shared memory for inter-process tensor communication. Mounting anemptyDirwithmedium: MemoryavoidsSIGBUSbus errors.initialDelaySeconds: 45: Prevents Kubernetes from marking the pod as failed while the model weights stream from storage into GPU memory.gpu-memory-utilization: 0.90: Reserves 90% of GPU VRAM for model weights and KV-cache allocations while leaving 10% headroom for memory fragmentation.
Should You Run AI on Kubernetes? Decision Framework
Before investing in a self-hosted Kubernetes AI platform, evaluate your organization’s workload requirements:
+-------------------------------------------------------------------------------+
| Kubernetes vs Managed AI Decision Matrix |
+-------------------------------------------------------------------------------+
USE MANAGED SERVERLESS APIS (Bedrock / Vertex / OpenAI / Together):
┌───────────────────────────────────────────────────────────────────────────┐
│ • Variable, sporadic inference traffic (< 50,000 requests/day) │
│ • Small engineering team with no dedicated Kubernetes or SRE specialists │
│ • Experimentation and proof-of-concept phase │
│ • Requirement for proprietary foundation models │
└───────────────────────────────────────────────────────────────────────────┘
RUN ON PRODUCTION KUBERNETES:
┌───────────────────────────────────────────────────────────────────────────┐
│ • Sustained, high-throughput request volume (> 500 tokens/sec sustained) │
│ • Strict data residency, HIPAA, or on-premise regulatory requirements │
│ • Custom fine-tuned weights, domain adapters, or proprietary architectures│
│ • Self-hosting delivers lower unit economics at high scale │
└───────────────────────────────────────────────────────────────────────────┘Frequently Asked Questions
Why does standard CPU autoscaling fail for AI inference on Kubernetes?
Standard Horizontal Pod Autoscalers (HPA) rely on CPU and memory percentages. AI inference engines like vLLM allocate nearly 100% of GPU VRAM upfront for KV cache buffers, making memory appear constant. Autoscaling must be driven by inference metrics like request queue depth (vllm:num_requests_waiting) using KEDA.
What is the difference between NVIDIA MIG and Time-Slicing?
NVIDIA Multi-Instance GPU (MIG) physically partitions an enterprise GPU into up to 7 hardware-isolated instances with dedicated memory and compute cores. Time-slicing shares a GPU over time without memory isolation, meaning one pod exceeding VRAM will cause all co-located pods to crash with OOM errors.
How can I avoid slow cold starts when scaling AI pods?
To mitigate cold starts: (1) cache model safetensors on host NVMe volumes using persistent CSI drivers or daemonset preloaders, (2) utilize model formats optimized for fast memory mapping, and (3) configure container images with pre-installed CUDA libraries.
What is PagedAttention in modern inference serving?
PagedAttention is an algorithm created by vLLM that manages the Key-Value (KV) cache memory like virtual memory pages in an operating system. It eliminates memory fragmentation, allowing inference servers to handle 2x to 4x higher request concurrency on the same GPU hardware.
When is it more cost-effective to run AI on Kubernetes vs SaaS APIs?
Self-hosting AI models on Kubernetes becomes cost-effective when request volume is consistently high and predictable (e.g., millions of daily tokens) or when privacy, data residency, and low latency mandate keeping data within a private cloud or on-premise VPC.