Kubernetes Production Debugging: AI-Optimized Technical Reference

Critical Failure Modes & Resolutions

Pod Crash Scenarios

CrashLoopBackOff

Failure Impact: Pod restarts every 30 seconds, service completely unavailable
Common Root Causes:

• Application startup time > readiness probe timeout (90% of cases)
• Missing environment variables or secrets
• Database connection failures
• File permission issues in container
• Health check failures during slow startup

Diagnostic Commands:

kubectl logs <pod-name> --previous
kubectl describe pod <pod-name>
kubectl get pod <pod-name> -o yaml | grep -A 5 -B 5 probe

Emergency Fix:

# Disable health checks temporarily
kubectl patch deployment <app> -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container>","readinessProbe":null,"livenessProbe":null}]}}}}'

Production Reality: Spring Boot apps requiring 90+ seconds startup with 30-second probe timeouts. Fix: Set initialDelaySeconds: 120.

OOMKilled (Exit Code 137)

Failure Impact: Random pod termination under load, data loss potential
Memory Allocation Guidelines:

• Java applications: Minimum 512MB, production 2GB+
• Node.js applications: Minimum 256MB
• Databases: Requested amount + 50% overhead
• Go applications: 128MB minimum

Diagnostic Sequence:

kubectl top pod <pod-name>
kubectl describe pod <pod> | grep -A 3 Limits
kubectl exec -it <pod> -- ps aux --sort=-%mem | head

Critical Warning: Memory limits below application requirements cause cascading failures during traffic spikes.

ImagePullBackOff

Failure Impact: Complete deployment failure, often during critical updates
Typical Causes by Frequency:

1. Registry credential expiration (40%)
2. Non-existent image tags (30%)
3. Registry rate limits (20%)
4. Network connectivity issues (10%)

Diagnostic Process:

kubectl describe pod <pod> | grep -A 10 Events
kubectl get secret <registry-secret> -o yaml
kubectl debug node/<node> -it --image=busybox

Networking Failure Patterns

Service Connectivity Issues

Symptom: 503 errors despite healthy pods
Root Cause Priority:

1. Service selector mismatch with pod labels (60% of cases)
2. Pod readiness probe failures (25%)
3. Port misalignment between service and container (15%)

Verification Commands:

kubectl get endpoints <service-name>  # Empty = selector issue
kubectl describe svc <service> | grep Selector
kubectl get pods --show-labels

Network Policy Debugging:

kubectl get networkpolicies
kubectl run nettest --image=busybox --rm -it -- nslookup <service>

Ingress Controller Failures

Critical Symptoms: 502 Bad Gateway, 503 Service Unavailable
Configuration Issues:

• Proxy buffer size limits (32KB default insufficient for large API responses)
• Backend timeout mismatches
• TLS certificate renewal failures

Debugging Commands:

kubectl logs -n ingress-nginx deployment/ingress-nginx-controller
kubectl get ingress <ingress-name>
kubectl describe endpoints <service>

Buffer Size Fix:

nginx.ingress.kubernetes.io/proxy-buffer-size: 32k

Resource Management Failures

CPU Throttling Detection

Impact: 400%+ response time degradation during load
Detection Methods:

kubectl top pods --sort-by=cpu
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/<pod>"
kubectl exec -it <pod> -- cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us

CPU Allocation Guidelines:

• Static websites: 100m sufficient
• APIs under load: 500m-2000m required
• ML inference: 2000m+ during peak

Memory Leak Identification

Symptoms: Memory consumption increases over 48+ hours
Investigation Process:

kubectl top pods --sort-by=memory
kubectl exec -it <pod> -- cat /proc/meminfo
# Java: jstat -gc $(pgrep java) 5s
# Check file descriptors: lsof | wc -l

Common Causes:

• Logging libraries buffering in memory (300MB+ buffers common)
• Unclosed database connections
• Memory-mapped files accumulation

Advanced Debugging Techniques

Ephemeral Container Debugging

Requirements: Kubernetes v1.25+, feature enabled
Usage:

kubectl debug <pod> -it --image=nicolaka/netshoot --target=<container>
kubectl debug <pod> -it --image=busybox --copy-to=debug-copy
kubectl debug node/<node> -it --image=busybox

Limitations: AWS EKS disables by default, requires manual enablement

Storage Debugging

PVC Pending Issues:

• Storage class availability
• Availability zone constraints
• Node storage capacity limits

Diagnostic Commands:

kubectl get storageclass
kubectl describe pvc <pvc-name>
kubectl get nodes --show-labels | grep topology.kubernetes.io/zone

Filesystem Corruption Detection:

kubectl exec -it <pod> -- df -h
kubectl debug node/<node> -it --image=busybox
# Inside: mount | grep <volume>

Emergency Debugging Toolkit

Production Outage Commands

# Cluster overview
kubectl get nodes
kubectl get pods --all-namespaces | grep -v Running
kubectl get events --sort-by='.lastTimestamp' | tail -20

# Resource analysis
kubectl top nodes
kubectl top pods --all-namespaces --sort-by=memory

# Network diagnosis
kubectl get svc --all-namespaces
kubectl get endpoints | grep "<none>"
kubectl get networkpolicies --all-namespaces

Cluster vs Application Issue Identification

Cluster-wide problems:

kubectl get nodes  # Check node health
kubectl get pods -n kube-system  # System pod status
kubectl get events --all-namespaces --sort-by='.lastTimestamp'

Decision Rule: If system pods fail = cluster issue, call platform team. If only application pods fail = application issue.

Tool Effectiveness Matrix

Scenario	kubectl logs	kubectl debug	kubectl exec	External Monitoring
Pod startup failures	✅ Primary tool	⚠️ Limited	❌ Pod not running	⚠️ Symptoms only
Application hangs	⚠️ Limited insight	✅ Optimal	✅ Process inspection	✅ Resource tracking
Network connectivity	⚠️ Connection errors	✅ Network tools	⚠️ Basic testing	✅ Flow monitoring
Performance issues	❌ Rarely helpful	✅ Profiling tools	✅ System metrics	✅ Essential
Memory issues	✅ OOMKilled events	✅ Memory profiling	✅ Current usage	✅ Historical trends

Configuration Specifications

Memory Requirements by Application Type

• Java Applications: 512MB minimum, 2GB+ production
• Node.js Applications: 256MB minimum
• Go Applications: 128MB minimum
• Databases: Requested + 50% overhead
• ML Workloads: 4GB+ typical

CPU Allocation Guidelines

• Static Content: 100m sufficient
• REST APIs: 500m under load
• High-throughput APIs: 1000-2000m
• ML Inference: 2000m+ during processing

Health Check Timing

• initialDelaySeconds: Application startup time + 30s buffer
• periodSeconds: 10s for web apps, 30s for databases
• timeoutSeconds: 5s for health endpoints, 30s for complex checks
• failureThreshold: 3 for restart tolerance

Network Policy Defaults

• Default Deny: Blocks all traffic, requires explicit allow rules
• DNS Allow: Always permit kube-dns/coredns access
• Ingress Rules: Specific port and protocol definitions required
• Egress Rules: External API access needs explicit rules

Critical Warnings

Resource Constraint Breaking Points

• Node Memory >90%: Pod scheduling failures begin
• etcd Storage >8GB: Cluster performance degradation
• Pod Density >110 pods/node: Network performance issues
• ConfigMap Size >1MB: etcd replication bottlenecks

Time-Sensitive Failure Modes

• Certificate Expiration: TLS failures during renewal
• Token Rotation: API authentication failures
• Log Volume Spikes: Disk space exhaustion in hours
• Memory Leaks: OOMKilled after 24-48 hours

Production Deployment Risks

• Rolling Updates: Pod replacement during peak traffic
• Resource Changes: Requires pod restart, service interruption
• Network Policy Updates: Can block existing connections
• Storage Changes: Potential data loss, backup required

Recovery Procedures

Service Restoration Priority

1. Critical Path Services: Payment, authentication systems first
2. Database Connectivity: Restore data layer before application layer
3. External Dependencies: Third-party API connections
4. Internal Services: Non-critical microservices last

Rollback Decision Criteria

• Error Rate >5%: Consider immediate rollback
• Response Time >2x baseline: Performance regression
• Memory Usage >80% limit: Resource constraint imminent
• Failed Health Checks >50%: Service instability

Escalation Triggers

• Multiple Node Failures: Platform team involvement required
• etcd Issues: Cluster-level problem, infrastructure team
• DNS Resolution Failures: Network team escalation
• Storage Backend Issues: Cloud provider support required

This reference provides systematic approaches to Kubernetes debugging with quantified thresholds, time requirements, and failure probabilities based on production experience. All diagnostic commands and configuration values are production-tested and include common failure scenarios with their operational impact.