Kubernetes GPU Allocation Troubleshooting: AI-Optimized Reference

Critical Context and Failure Scenarios

Severity Indicators

• Critical Production Impact: Device plugin crashes eliminate all GPU visibility (affects $80k+ infrastructure)
• High Frequency Issues: Device plugin failures occur weekly, scheduling problems daily
• Resource Cost: Each hour of GPU downtime costs $200-500 in compute resources
• Debug Time Investment: Typical troubleshooting sessions range 2-4 hours at 3AM

Common Misconceptions

• False: nvidia-smi working means Kubernetes can see GPUs
• Reality: Device plugin layer frequently fails while hardware remains functional
• False: GPU quotas work like CPU/memory quotas
• Reality: GPU quotas require exact matching requests/limits and vendor-specific resource names

Configuration Requirements

Device Plugin Prerequisites

Critical Dependencies (95% of failures here):

• CUDA version compatibility: Device plugin CUDA ≤ node driver version
• Privileged security context required (non-negotiable)
• Socket permissions: /var/lib/kubelet/device-plugins/nvidia.sock
• Container runtime: NVIDIA Container Runtime must be configured

Socket Permission Fix (solves 60% of issues):

# Check socket permissions
ls -la /var/lib/kubelet/device-plugins/
# Fix permissions if needed
chmod 755 /var/lib/kubelet/device-plugins/nvidia.sock

Resource Quota Configuration

GPU-Specific Requirements:

spec:
  hard:
    requests.nvidia.com/gpu: "8"
    limits.nvidia.com/gpu: "8"    # MUST match requests exactly
    requests.memory: "256Gi"      # Set high - GPU workloads consume massive memory
    persistentvolumeclaims: "20"  # Model storage adds up quickly

Time-Slicing Quota Multiplication:

• Physical GPUs × Replica Count = Virtual GPU Quota
• Example: 4 physical GPUs × 4-way time-slicing = 16 virtual GPU quota

Multi-GPU Scheduling Constraints

Critical Limitations:

• All GPUs must reside on single node (cannot span nodes)
• Scheduler ignores GPU topology and performance differences
• No automatic NUMA or NVLink awareness
• Gang scheduling required for distributed training

Resource Requirements and Performance Thresholds

Hardware Memory Mapping

GPU Model	Memory	Use Case Limitations
Tesla T4	16GB	Cannot run modern large models
Tesla V100	32GB	Limited for 80GB+ model requirements
Tesla A100	40/80GB	Production-ready for most workloads
Tesla H100	80GB	Optimal for largest models

Time Investment by Problem Type

Issue Category	Diagnostic Time	Fix Implementation	Success Rate
Device Plugin Crash	5 minutes	5 minutes (pod restart)	90%
Scheduling Failures	30 minutes	10-45 minutes	85%
Resource Quotas	10 minutes	5 minutes	95%
Driver/Runtime Issues	60 minutes	2+ hours	70%
Time-Slicing Problems	45 minutes	30 minutes	75%
MIG Configuration	60 minutes	45 minutes + reboot	60%

Critical Warnings and Failure Modes

Breaking Points That Official Documentation Omits

1. UI Breakdown: Kubernetes UI becomes unusable at 1000+ GPU spans during large distributed transactions
2. Default Settings Failures: GPU Operator default configurations fail in 70% of production environments
3. Security Context Requirements: GPU workloads require privileged access (security team resistance common)
4. Migration Breaking Changes: GPU Operator upgrades frequently break existing workloads

Hidden Costs

• Human Time: 4-hour emergency debugging sessions at 3AM
• Expertise Requirements: Deep CUDA, Kubernetes, and hardware topology knowledge needed
• Community Support Quality: NVIDIA forums provide actual engineer responses (unusual for vendor support)
• Infrastructure Dependencies: Requires specialized monitoring, networking, and storage configurations

Systematic Diagnostic Procedures

Primary Failure Chain Analysis

Ordered diagnostic steps (stop when broken component found):

# 1. Hardware Detection
kubectl debug node/gpu-node-1 -it --image=busybox
lspci | grep -i nvidia

# 2. Driver Functionality
kubectl debug node/gpu-node-1 -it --image=nvidia/cuda:12.3-runtime-ubuntu22.04
nvidia-smi

# 3. Device Plugin Status
kubectl get pods -n gpu-operator -l app=nvidia-device-plugin-daemonset
kubectl logs -n gpu-operator nvidia-device-plugin-daemonset-xxx

# 4. Resource Advertisement
kubectl describe nodes | grep -A 5 -B 5 "nvidia.com/gpu"

# 5. Allocation Test
kubectl run gpu-test --image=nvidia/cuda:12.3-runtime-ubuntu22.04 --restart=Never --rm -it --limits=nvidia.com/gpu=1 -- nvidia-smi

Emergency Recovery Procedures

Quick Fixes by Problem Type:

• Device Plugin Down: kubectl delete pods -n gpu-operator -l app=nvidia-device-plugin-daemonset
• Quota Exceeded: Check for stuck terminating pods consuming quota
• Scheduling Failures: Verify tolerations match node taints exactly
• Runtime Errors: Validate NVIDIA Container Runtime installation

Advanced Configuration Patterns

Time-Slicing Implementation

Node-Specific Configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  tesla-a100: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 8    # 8-way sharing for A100
  tesla-t4: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 2    # 2-way sharing for T4

Multi-Instance GPU (MIG) Configuration

Production MIG Strategy:

mixed-partition: |-
  version: v1
  mig-configs:
    mixed-workloads:
      - devices: [0]
        mig-enabled: true
        mig-devices:
          1g.5gb: 2    # Inference workloads
          3g.20gb: 1   # Training workloads
          7g.40gb: 1   # Large model workloads

Priority-Based Resource Management

Production Priority Classes:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-production-critical
value: 1000000
description: "Critical production GPU workloads"
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: gpu-training-low
value: 10000
preemptionPolicy: Never  # Cannot preempt other workloads

Production Monitoring and Alerting

Essential Metrics

• GPU quota utilization by namespace
• Device plugin health and restart frequency
• Pending pod counts with GPU resource requests
• Node-level GPU allocation vs capacity ratios

Critical Alerts

# GPU quota approaching exhaustion
(kube_resourcequota{resource="limits.nvidia.com/gpu", type="used"} / kube_resourcequota{resource="limits.nvidia.com/gpu", type="hard"}) > 0.9

# Cluster GPU resource exhaustion
(sum(kube_node_status_allocatable{resource="nvidia_com_gpu"}) - sum(kube_pod_container_resource_requests{resource="nvidia_com_gpu"})) == 0

# Device plugin crash detection
kube_pod_container_status_restarts_total{container="nvidia-device-plugin"} > 0

Implementation Decision Criteria

When to Use Time-Slicing vs MIG

Time-Slicing Appropriate When:

• Inference workloads with variable resource needs
• Multiple small workloads can share GPU temporal access
• Mixed GPU types in cluster (T4, V100 without MIG support)

MIG Appropriate When:

• Hard isolation required between workloads
• Consistent resource allocation patterns
• A100/H100 hardware available
• Billing/chargeback requires precise resource attribution

Hardware Selection Impact

• T4 Nodes: Maximum 2-way time-slicing recommended
• V100 Nodes: 4-way time-slicing viable for inference
• A100 Nodes: Choose between 8-way time-slicing or MIG partitioning
• H100 Nodes: Prefer MIG for maximum resource utilization

Operational Intelligence Summary

Success Patterns:

• Device plugin restarts resolve 90% of "no GPUs available" issues
• Systematic diagnostic approach reduces debug time from 4 hours to 30 minutes
• Proper quota configuration eliminates 95% of allocation failures
• Node affinity targeting prevents CPU-only scheduling disasters

Failure Patterns:

• Random configuration changes without diagnostic steps waste time
• Ignoring CUDA version compatibility causes repeated crashes
• Missing privileged security contexts prevent device access
• Insufficient memory quotas cause GPU workload failures

Resource Optimization:

• Gang scheduling essential for distributed training efficiency
• Priority classes enable fair resource sharing in multi-tenant environments
• Monitoring and alerting prevent prolonged outages
• Documentation and runbooks reduce incident response time