Debug Kubernetes Issues - The 3AM Production Survival Guide

Your Kubernetes cluster isn't failing because of some exotic edge case described in a 200-page white paper. It's failing because someone forgot to set memory limits, your ingress controller shit the bed, or AWS decided to randomly restart your nodes. Here's what actually breaks in the real world.

The Five Horsemen of Kubernetes Apocalypse

1. Pod Death Spiral (CrashLoopBackOff Hell)

What it looks like: Your pod keeps restarting every 30 seconds and kubectl describe shows CrashLoopBackOff status.

Why it happens:

• Your app crashes on startup but the container image doesn't fail properly
• Database connection strings are wrong (check your secrets)
• Missing environment variables that your app needs to start
• File permissions are fucked in your container (security context issues)
• Your health check is failing because the app takes 45 seconds to start but your readinessProbe times out after 10

The 5-minute fix:

## See what's actually happening
kubectl logs your-broken-pod --previous
kubectl describe pod your-broken-pod

## Check if it's a startup timing issue
kubectl get pod your-broken-pod -o yaml | grep -A 5 -B 5 probe

## Nuclear option: disable health checks temporarily
kubectl patch deployment your-app -p '{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"your-container\",\"readinessProbe\":null,\"livenessProbe\":null}]}}}}'

Real failure story: Our Spring Boot app took 90 seconds to start because it was downloading Maven dependencies on startup (brilliant architecture choice). The readiness probe was configured for 30 seconds. Pods kept getting killed just as they were about to be ready. Solution: increased initialDelaySeconds to 120 and fixed the Docker image to include dependencies. Took down prod for 2 hours because we "didn't want to change the health check configuration." Learn from our debugging mistakes.

2. OOMKilled - The Memory Massacre

What it looks like: kubectl get pods shows your pod restarted with exit code 137, and kubectl describe pod reveals the dreaded Reason: OOMKilled.

Why it happens:

• You allocated 128MB but your Java app needs 2GB (classic resource management failure)
• Memory leak in your application that accumulates over days
• No memory limits set, so your pod consumed everything until the node kernel murdered it (node resource management)
• Your container is running multiple processes and you only accounted for one (best practices guide)

The actual debugging process:

## Check current memory usage 
kubectl top pod your-pod-name
kubectl top node

## See memory limits vs requests
kubectl describe pod your-pod | grep -A 3 -B 3 Limits

## Check what's actually using memory in the container
kubectl exec -it your-pod -- ps aux --sort=-%mem | head

## Historical memory usage (if you have metrics-server)
kubectl get --raw \"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/your-pod\"

What actually worked: Doubled the memory limit from 256Mi to 512Mi, added memory monitoring, and discovered our logging library was buffering 300MB of logs in memory before writing to disk. The "temporary" fix became permanent because nobody had time to optimize the logger during Black Friday prep season.

3. ImagePullBackOff - The Registry Nightmare

What happens: Your pod sits in Pending state forever with ImagePullBackOff error, usually at the worst possible time.

Why it always happens:

• Registry is down (Docker Hub limits hit you)
• Wrong image tag (someone deployed v2.1.4 but it doesn't exist)
• Authentication failed (registry credentials expired)
• Image is too big for the node (8GB image on a 10GB node)
• Private registry URL is wrong (happens during migrations)

Debug it properly:

## Check the actual error message
kubectl describe pod your-pod | grep -A 10 Events

## Test image pull manually
kubectl debug node/your-node -it --image=busybox
crictl pull your-registry/your-image:tag

## Check if it's an auth issue
kubectl get secret your-registry-secret -o yaml
echo \"base64-string\" | base64 -d | jq .

Production war story: During a midnight deployment, our image pull started failing with "unauthorized" errors. Turns out the CI/CD system was using an API token that expired after 90 days. The image built fine, pushed fine, but pulling failed because the cluster used different credentials. Spent 3 hours debugging container registries while our app was down. The fix was rotating a single API key that should have been automated 6 months earlier.

The Networking Black Hole

Services That Pretend to Work

The symptoms: kubectl get svc shows your service exists, but curl returns connection refused, and your ingress returns 503 errors.

What's actually broken:

## Check if service has endpoints
kubectl get endpoints your-service

## No endpoints? Your selector is wrong
kubectl get pods --show-labels
kubectl describe svc your-service | grep Selector

## Endpoints exist but connection fails? Check network policies
kubectl get networkpolicies
kubectl describe netpol your-policy

## Still broken? Test from inside the cluster
kubectl run debug-pod --image=busybox --rm -it -- sh
## Inside the pod:
nslookup your-service
telnet your-service 80

Reality check: Network policies are the reason your microservices can't talk to each other. Someone enabled "secure by default" policies that block everything, then quit without documenting which services need to communicate. You'll spend hours adding network policy exceptions one by one while your staging environment is completely broken.

Ingress Controllers Having Existential Crises

What you see: External users get 502 Bad Gateway, 503 Service Unavailable, or timeouts.

What's actually happening:

## Check ingress controller logs (it's probably nginx-ingress)
kubectl logs -n ingress-nginx deployment/ingress-nginx-controller

## Check if ingress has an IP
kubectl get ingress your-ingress

## Verify backend services are healthy
kubectl get pods -l app=your-app
kubectl describe endpoints your-service

## Test internal service connectivity
kubectl port-forward svc/your-service 8080:80
curl localhost:8080

War story: Our ingress worked fine for months, then started returning 502 errors during high traffic. The nginx-ingress logs showed upstream sent too big header while reading response header from upstream. Our API was returning 32KB JSON responses, but nginx proxy buffer was set to 4KB. The solution was adding this annotation: nginx.ingress.kubernetes.io/proxy-buffer-size: 32k. Took down our API for 4 hours because nobody knew ingress controllers had buffer limits.

Resource Starvation - When Everything Fights for CPU

The death spiral: Pods get CPU throttled, become slow, health checks fail, pods restart, rinse and repeat.

Debug resource issues:

## Check current resource usage
kubectl top nodes
kubectl top pods --sort-by=cpu
kubectl top pods --sort-by=memory

## Find pods without resource limits
kubectl get pods -o jsonpath=\"{\.items[*].spec.containers[*].resources}\" | jq .

## Check for CPU throttling
kubectl describe pod your-pod | grep -A 5 -B 5 throttl

The fix that actually worked: Set proper CPU limits based on actual usage, not guesses. Monitor for a week first. Yes, it takes time. No, you can't wing it. CPU limits of 100m work for static websites, not for your machine learning inference API that needs 2000m during peak load.

Your cluster is probably broken right now for one of these reasons. The next section covers the specific commands and debugging workflows that actually fix these problems instead of just identifying them.

When `kubectl logs` and `kubectl describe` aren't enough to solve your production nightmare, you need the advanced arsenal. These are the debugging techniques that separate engineers who panic from those who methodically debug complex cluster failures.

Ephemeral Containers - Debug Without Destroying Production

The game changer: Ephemeral containers let you inject debugging tools into running pods without modifying your production images.

## Add a debugging container to a running pod
kubectl debug misbehaving-pod -it --image=nicolaka/netshoot --target=my-app

## Once inside:
## - tcpdump to capture network traffic
## - dig to test DNS resolution  
## - curl to test HTTP endpoints
## - ss to check listening ports

Real debugging session: Our payment API was randomly timing out during checkout. Using ephemeral containers, we discovered the database connection pool was exhausted because the connection timeout was set to 30 seconds but queries were taking 45 seconds. Fixed by increasing the timeout configuration and optimizing the slow queries through performance monitoring.

Limitations:

• Requires Kubernetes v1.25+ with ephemeral containers enabled
• AWS EKS disables this by default (enable via feature gate)
• Some managed K8s services don't support it yet

Alternative for older clusters: Create debugging pods with shared volumes:

kubectl run debug-pod --image=busybox --rm -it --overrides='{\"spec\":{\"shareProcessNamespace\":true}}'

Deep Dive: Networking Debug Hell

When Services Work But Applications Don't

The invisible problem: Your service endpoints look healthy, pods are ready, but requests are failing intermittently.

## Check if load balancing is working correctly
kubectl get endpoints your-service -o yaml

## Test direct pod connectivity (bypass service)
POD_IP=$(kubectl get pod your-pod -o jsonpath='{.status.podIP}')
kubectl run test --image=busybox --rm -it -- wget -O- http://\$POD_IP:8080/health

## Check iptables rules (advanced)
kubectl debug node/worker-node -it --image=busybox
## Inside node container:
iptables -t nat -L | grep your-service

What we found: Half our pods were healthy, half were failing health checks due to a database migration that only affected certain database shards. The service was load-balancing between healthy and unhealthy pods, causing 50% of requests to fail. Fixed by implementing better health checks that actually tested database connectivity.

Ingress Debugging: When External Traffic Doesn't Reach Your Pods

The full debugging chain:

## 1. Verify ingress has external IP
kubectl get ingress your-ingress -o wide

## 2. Check ingress controller logs
kubectl logs -n ingress-nginx deployment/ingress-nginx-controller --tail=100 -f

## 3. Test ingress rule parsing
kubectl get ingress your-ingress -o yaml | grep -A 5 -B 5 rules

## 4. Verify TLS certificates (if using HTTPS)
kubectl describe secret your-tls-secret
openssl x509 -in <(kubectl get secret your-tls-secret -o jsonpath='{.data.tls\.crt}' | base64 -d) -text -noout

Production nightmare: Our ingress was routing traffic correctly during the day but failing at night. Investigation revealed the TLS certificate was valid, but the ingress controller was failing SSL handshakes after midnight. The root cause: our certificate renewal job ran at 12:01 AM and briefly replaced the certificate with an invalid one during the renewal process. Fixed by adding proper certificate validation to the renewal script and updating certificates during low-traffic hours.

Storage Debugging: When Persistent Volumes Become Persistent Problems

PVC Stuck in Pending State

The typical causes:

## Check if storage class exists
kubectl get storageclass

## Verify PVC configuration
kubectl describe pvc your-pvc

## Check available storage on nodes
kubectl describe nodes | grep -A 5 -B 5 \"Allocated resources\"

## For EBS/cloud storage, check zone constraints
kubectl get nodes --show-labels | grep topology.kubernetes.io/zone
kubectl get pv | grep Available

What actually happened: Our PVC was requesting 100GB in us-west-2a, but all available PVs were in us-west-2b. Kubernetes couldn't schedule the pod because of availability zone constraints. The fix required either creating cross-zone PVs or ensuring pod affinity matched PV zones.

Corrupted Persistent Volumes

When your data is there but not accessible:

## Check mount status inside the container
kubectl exec -it your-pod -- df -h
kubectl exec -it your-pod -- ls -la /var/data/

## Check volume mount events
kubectl describe pod your-pod | grep -A 10 -B 10 Volume

## For filesystem corruption, check node-level mounts
kubectl debug node/your-node -it --image=busybox
## Inside: mount | grep your-volume

War story: Our PostgreSQL pod was crashing with "database system was interrupted" errors. The PV was mounted correctly, but filesystem checks revealed corruption. The underlying EBS volume had experienced an unexpected detachment during an AWS maintenance event. We had to restore from backup because the filesystem corruption was too extensive to repair. Lesson: Always test your backup restoration process before you need it.

Resource Analysis: Understanding What's Really Consuming Resources

CPU Throttling Detection

Beyond basic kubectl top:

## Check for CPU throttling in pod metrics
kubectl get --raw \"/apis/metrics.k8s.io/v1beta1/namespaces/default/pods/your-pod\" | jq .

## Historical throttling (if using Prometheus)
## container_cpu_cfs_throttled_seconds_total metric shows throttling events

## Check cgroup limits inside container
kubectl exec -it your-pod -- cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
kubectl exec -it your-pod -- cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us

The discovery: Our API response times increased by 400% during peak load. CPU metrics showed normal usage, but deeper investigation revealed severe CPU throttling. The pods were hitting their 100m CPU limits during request processing, causing response times to degrade. Fixed by increasing CPU limits to 500m and implementing request queuing.

Memory Leak Detection

Advanced memory analysis:

## Get detailed memory breakdown
kubectl exec -it your-pod -- cat /proc/meminfo

## For Java applications, check heap usage
kubectl exec -it your-pod -- jstat -gc $(pgrep java) 5s

## Memory usage trends (requires monitoring)
kubectl top pods --sort-by=memory | head -20

## Check for memory-mapped files consuming space
kubectl exec -it your-pod -- find /proc/*/maps -exec cat {} \; | grep -v \"00000000-00000000\" | wc -l

Real debugging scenario: Our Node.js application was consuming 2GB+ memory after running for 48 hours, despite having a memory limit of 512MB. The investigation revealed our logging library was keeping file descriptors open and buffering logs in memory. We had 50,000+ open file handles consuming memory. Fixed by implementing proper log rotation and reducing log buffer sizes.

Cluster-Level Debugging: When the Platform is the Problem

etcd Performance Issues

When the brain is slow:

## Check etcd health (from control plane)
kubectl get componentstatuses
etcdctl --endpoints=https://etcd-server:2379 endpoint health

## Monitor etcd performance
etcdctl --endpoints=https://etcd-server:2379 endpoint status --write-out=table

## Check for large objects in etcd
kubectl get --raw=\"/api/v1/namespaces/default/configmaps\" | jq '.items[] | select(.metadata.name==\"large-config\") | .data | length'

The incident: Our entire cluster became unresponsive during a deployment. Investigation showed etcd was running out of disk space due to a ConfigMap containing 50MB of JSON data. The large ConfigMap was being replicated across all etcd nodes, causing disk I/O bottlenecks. Fixed by moving large configurations to external storage and implementing ConfigMap size limits.

Node Resource Exhaustion

When nodes stop accepting new pods:

## Check node conditions
kubectl describe nodes | grep -A 10 Conditions

## Node resource allocation
kubectl describe nodes | grep -A 10 \"Allocated resources\"

## System resource usage on nodes
kubectl debug node/your-node -it --image=busybox
## Inside: top, free -h, df -h

The pattern: During traffic spikes, our nodes would reach 90% memory utilization and stop scheduling new pods. The issue wasn't application memory usage but kernel memory consumption due to too many network connections. We had to tune kernel parameters and implement connection limits at the ingress level.

Understanding these advanced techniques separates operational firefighting from systematic problem-solving. The next section covers how to prevent these issues before they happen in production.