When Your Pod Won't Stop Dying - Deep Debugging for the Shit That Actually Matters

Your pod is still dying and you've tried everything obvious. Memory limits? Set. Environment variables? Checked twice. Health checks? Perfect. Yet here you are at 2am watching "CrashLoopBackOff" and "Back-off 5m0s restarting failed container" like some twisted Groundhog Day.

Here's what actually happens: your code's fine, your manifest's fine, but some invisible cluster bullshit is murdering your pods. Not the obvious stuff like memory limits - the weird node constraints, storage backend timeouts, or runtime security policies that don't show up anywhere.

Been debugging this shit for years and it's always the same story: works on laptop, passes CI, manifest looks good, pod dies every 30 seconds anyway.

Node Scheduling Conflicts That Kill Pods Silently

Taints and tolerations - Kubernetes's passive-aggressive way of fucking with you. Pod starts fine then crashes when it tries to do actual work because some node has taints you didn't know about.

## Check node taints that might affect scheduling
kubectl describe nodes | grep -A 5 -B 5 \"Taints\"

## Examine specific node taints and their effects
kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}'

## Check if your pods have appropriate tolerations
kubectl describe pod <pod-name> | grep -A 10 \"Tolerations\"

Node affinity rules can create scenarios where pods start but fail during runtime due to resource constraints or environment mismatches. A pod scheduled on an inappropriate node might lack GPU access, specific storage types, or network configurations required for operation.

## Check node labels and affinity rules
kubectl get nodes --show-labels

## Examine pod affinity constraints
kubectl describe pod <pod-name> | grep -A 15 \"Node-Selectors\\|Affinity\"

## Verify resource availability on assigned nodes
kubectl describe node <node-name> | grep -A 10 \"Allocated resources\"

Spent way too long on one where ML training kept dying with "CUDA error: no device found". Made no sense - we had GPU nodes, memory was fine, everything looked right.

Turned out our YAML was missing some GPU selector bullshit so Kubernetes kept scheduling GPU pods on regular nodes. I don't even remember the exact fix now, just that it was something stupid with node selectors.

Container Runtime and Node-Level Storage Issues

Container runtime problems are the worst kind of debugging hell because the error messages are useless and Google searches return nothing helpful. You'll spend hours checking containerd configuration, CRI-O settings, or Docker daemon issues while your pod dies with cryptic exit codes.

Security contexts get fucked up and the runtime has no idea what to do, or filesystem permissions are broken but only when your app tries to actually write something.

## Check container runtime logs on the node
kubectl get pod <pod-name> -o wide  # Get node name
kubectl describe node <node-name> | grep \"Container Runtime\"

## Examine kubelet logs on the problematic node (requires node access)
sudo journalctl -u kubelet -f --since \"1 hour ago\"

## Check for runtime-specific errors
sudo crictl ps -a | grep <container-name>
sudo crictl logs <container-id>

Persistent volume mounting failures happen when your pod looks fine but crashes the second it tries to read or write files. The volume mounts look perfect in kubectl but the storage backend is having a meltdown.

## Check PV and PVC status for mounting issues
kubectl get pv,pvc

## Examine volume mount details in the pod
kubectl describe pod <pod-name> | grep -A 20 \"Volumes\\|Mounts\"

## Check for volume-related events
kubectl get events --field-selector involvedObject.name=<pod-name> | grep -i volume

Storage backend issues cause the most infuriating failures - your pod runs fine for a few minutes then dies when the network storage decides to time out or the cloud storage provisioner shits the bed.

Network Policy and Service Mesh Configuration Conflicts

Network policies are silent killers. Your pod starts perfectly, passes health checks, then crashes the moment it tries to call another service because some network policy is quietly blocking the connection.

I've seen this kill entire microservice deployments where developers spend days debugging "connection refused" errors, not realizing that Calico or Cilium policies are dropping their traffic.

## Check network policies that might affect your pod
kubectl get networkpolicies -A

## Examine specific policy rules
kubectl describe networkpolicy <policy-name> -n <namespace>

## Test network connectivity from the failing pod
kubectl exec <pod-name> -- nc -zv <target-service> <port>
kubectl exec <pod-name> -- nslookup <service-name>

Service mesh bullshit (Istio, Linkerd, Consul Connect) loves to crash your app when the sidecar proxy decides to intercept traffic in stupid ways. Connection timeouts, TLS cert failures, routing fuckups that break your startup sequence.

## Check for service mesh sidecar injection
kubectl describe pod <pod-name> | grep -A 5 -B 5 \"istio\\|linkerd\\|consul\"

## Examine sidecar proxy logs
kubectl logs <pod-name> -c istio-proxy
kubectl logs <pod-name> -c linkerd-proxy

## Verify service mesh configuration
istioctl proxy-config cluster <pod-name>
linkerd stat pod <pod-name>

Resource Quota and Limit Range Enforcement Issues

Resource quotas - silent killers that let your pod start then murder it later. Pod runs for a few minutes, then gets OOMKilled by some namespace quota you didn't know existed. Error messages won't tell you this - you have to dig through cluster events like a detective.

## Check resource quotas affecting your namespace
kubectl describe resourcequota -n <namespace>

## Examine namespace-level resource usage
kubectl describe namespace <namespace>

## Check for resource-related events
kubectl get events -A | grep -i \"quota\\|limit\"

Limit ranges are sneaky fuckers that apply hidden limits to your pods. Your app crashes when it hits these secret constraints that don't show up anywhere obvious in your pod spec.

## Check limit ranges in your namespace
kubectl describe limitrange -n <namespace>

## Compare pod resource requests with limit ranges
kubectl describe pod <pod-name> | grep -A 10 \"Limits\\|Requests\"

Advanced Node Health and Kernel-Level Issues

Node-level resource exhaustion gets weird fast. Your pod dies but kubectl top node shows plenty of CPU and memory available. Turns out the node ran out of inodes, or hit some obscure kernel limit on network connections, or the disk I/O subsystem is having a breakdown.

## Check detailed node resource usage
kubectl top node <node-name>

## Examine node conditions for health issues
kubectl describe node <node-name> | grep -A 10 \"Conditions\"

## Check for node pressure conditions
kubectl get nodes -o wide
kubectl describe node <node-name> | grep -i \"pressure\\|full\"

Kernel parameter bullshit kills apps that try to open too many connections, file handles, or spawn too many processes. Your code is fine, the kernel just decided to be a dick about resource limits.

When your pod keeps dying despite perfect config, it's always invisible infrastructure bullshit. Node taints, storage backends timing out, network policies written by someone who quit three years ago.

Accept that your app is probably fine and the cluster is lying. Check node health, storage status, runtime logs - don't waste more time staring at your code.

But when cluster-level debugging still doesn't show shit? When kubectl says everything's healthy but your pod dies every few minutes anyway? Time for the nuclear options.

Your pod is still dying after 6 hours of debugging and you're ready to set the whole cluster on fire. Time to break out the big guns. When kubectl describe and kubectl logs give you nothing useful, you need system-level debugging, container runtime analysis, and kernel investigation tools that actually show you what's happening.

These techniques require cluster admin access you probably don't have, but when you get it, they'll expose the invisible fuckery that's killing your pods.

System Call Tracing and Low-Level Debugging

strace is your best friend when application logs are lying to you. It shows every system call your app makes before it dies, which is infinitely more useful than "Error: something went wrong" in your logs.

I've used strace to debug everything from file permission issues to network connection failures that didn't show up anywhere else. It's the debugging equivalent of putting on X-ray glasses.

## Attach strace to a running container process (requires privileged access)
kubectl exec <pod-name> -- strace -f -e trace=all -o /tmp/strace.out <your-application-command>

## Monitor system calls during container startup
kubectl debug <pod-name> --image=nicolaka/netshoot --target=<container-name>
## Inside the debug container:
strace -f -e trace=open,openat,read,write,connect -p <pid>

Filter strace or you'll drown in garbage (like 50,000 lines of useless syscalls):

## Network calls for connection bullshit
strace -e trace=network -f <command>

## File calls for permission fuckery  
strace -e trace=file -f <command>

## Memory calls for allocation failures
strace -e trace=memory -f <command>

Had this Node.js thing dying with "ENOENT: no such file or directory" but the files were RIGHT THERE when I'd ls them. Spent days thinking it was volume mounts.

strace showed the app looking for Config.json but the file was config.json. Think the volume mount was case-insensitive but Node was case-sensitive? Only failed when some cache was cold. Took forever to figure out something that stupid.

Container Runtime Deep Dive

crictl is what you use when kubectl lies. The container runtime often knows exactly why your container died, but kubectl doesn't bother telling you.

crictl talks directly to containerd or CRI-O and gives you the real story about what happened to your container. It's like having a direct line to the thing that's actually running your code.

## Get container runtime details
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].containerID}'

## For containerd runtime (most common in modern clusters)
sudo crictl ps -a | grep <container-name>
sudo crictl inspect <container-id>
sudo crictl logs <container-id>

## Examine container runtime events
sudo crictl events | grep <container-id>

Runtime security restrictions (seccomp, AppArmor, SELinux) can cause applications to crash when attempting forbidden system operations:

## Check if seccomp profiles are blocking system calls
kubectl describe pod <pod-name> | grep -A 5 -B 5 seccomp

## Examine AppArmor restrictions
kubectl describe pod <pod-name> | grep -A 5 -B 5 apparmor

## Check for SELinux denials (on RHEL/CentOS nodes)
sudo ausearch -m AVC -ts recent | grep <container-name>

Kernel-Level Resource Investigation

Process limits and resource exhaustion at the kernel level often cause crashes that appear as application errors:

## Check process limits within the container
kubectl exec <pod-name> -- cat /proc/self/limits

## Monitor resource usage in real-time
kubectl exec <pod-name> -- cat /proc/meminfo
kubectl exec <pod-name> -- cat /proc/loadavg

## Check for OOM events in kernel logs (requires node access)
sudo dmesg | grep -i "killed process\|out of memory"

File descriptor exhaustion causes crashes when applications can't open new files or network connections:

## Check file descriptor usage
kubectl exec <pod-name> -- ls /proc/self/fd | wc -l
kubectl exec <pod-name> -- ulimit -n

## Monitor file descriptor leaks
kubectl exec <pod-name> -- lsof -p <pid> | wc -l

Network Stack Deep Debugging

Network namespace investigation reveals connectivity issues that standard network tests miss:

## Enter the pod's network namespace for detailed debugging
kubectl debug <pod-name> --image=nicolaka/netshoot --target=<container-name>

## Inside the debug container, examine network configuration
ip addr show
ip route show
iptables -L -n
ss -tuln  # Check listening ports
netstat -i  # Interface statistics

DNS resolution deep dive exposes subtle DNS failures that cause application crashes:

## Trace DNS queries step by step
kubectl exec <pod-name> -- strace -e trace=connect,sendto,recvfrom -f dig <hostname>

## Check DNS server accessibility
kubectl exec <pod-name> -- nc -zv <dns-server-ip> 53
kubectl exec <pod-name> -- tcpdump -i any port 53

Connection tracking and firewall rules can silently drop connections, causing applications to crash during network operations:

## Check connection tracking table (requires privileged access)
kubectl exec <pod-name> -- conntrack -L
kubectl exec <pod-name> -- conntrack -S  # Statistics

## Examine netfilter rules
kubectl exec <pod-name> -- iptables-save | grep <target-service>

Advanced Monitoring and Observability

Kubernetes audit logs reveal cluster-level operations that might affect your pod:

## Check audit logs for pod-related events (requires cluster admin access)
kubectl get events --all-namespaces | grep <pod-name>

## Examine API server audit logs for RBAC or admission controller issues
sudo grep <pod-name> /var/log/audit/audit.log

Extended pod diagnostics using kubectl debug with advanced debugging containers:

## Use specialized debugging images with advanced tools
kubectl debug <pod-name> --image=registry.k8s.io/pause:3.9 --copy-to=debug-copy
kubectl exec debug-copy -- ps aux  # See all processes
kubectl exec debug-copy -- env      # Check environment variables
kubectl exec debug-copy -- mount    # Examine mounted filesystems

Performance and Resource Profiling

Application profiling within the Kubernetes environment reveals performance issues that cause crashes under load:

## CPU profiling for performance-related crashes
kubectl exec <pod-name> -- perf record -g -p <pid>
kubectl exec <pod-name> -- perf report

## Memory profiling to identify leaks
kubectl exec <pod-name> -- valgrind --tool=memcheck <your-application>

I/O and disk performance analysis identifies storage-related failures:

## Monitor I/O operations
kubectl exec <pod-name> -- iotop -p <pid>
kubectl exec <pod-name> -- iostat -x 1

## Check filesystem health
kubectl exec <pod-name> -- df -i    # Inode usage
kubectl exec <pod-name> -- lsof +D /app  # Open files in directory

Systematic Elimination Process

After 8 hours of this garbage, stop randomly trying fixes and debug systematically:

1. Isolate variables: Create a minimal reproduction case with the same base image but simplified configuration
2. Binary search configuration: Systematically enable/disable configuration options to identify the problematic setting
3. Runtime comparison: Compare working vs. failing environments at the system call level using strace
4. Stress testing: Apply controlled load to identify race conditions or resource exhaustion patterns
5. Timeline analysis: Correlate crashes with cluster events, resource usage spikes, or external dependencies

Reality check: Security will never give you admin access for these tools. But when prod's down at 3am and the CEO's asking questions, suddenly you get root and these tools show exactly what's murdering your pods.

The key is being systematic instead of randomly trying shit. Start with strace, move to crictl, then dig into kernel logs and network debugging. One of these layers will tell you the truth.

Once you know what's actually failing, the fix is usually straightforward. It's the finding out that takes forever.

After years of this shit, you start seeing patterns. Same weird edge cases, same gotchas that shouldn't be possible but happen anyway.

Even with all these tools you'll still hit scenarios that make no sense. When you've tried strace, crictl, kernel logs, everything - that's when you question your life choices.