Your Kubernetes Cluster is Down at 3am: Now What?

When everything's on fire, you have maybe 5 minutes before panic sets in and management starts breathing down your neck. Here's what I learned after getting paged at 3am more times than I care to remember.

Step 1: Is kubectl Even Working?

Before you do anything fancy, check if you can talk to your cluster at all:

kubectl cluster-info

If this times out, your control plane is fucked. I wasted 4 hours once debugging a "cluster failure" that was just my VPN disconnecting. Always check the obvious shit first.

The kubectl cluster-info command should return URLs for your API server and other core services. If you're getting timeouts, check your kubeconfig file and network connectivity before diving deeper.

Common error messages you'll actually see:

• Unable to connect to the server: dial tcp: lookup kubernetes.docker.internal - Your kubeconfig is pointing to the wrong cluster
• The connection to the server localhost:8080 was refused - You forgot to set your context
• error: You must be logged in to the server (Unauthorized) - Your token expired while you were sleeping
• dial tcp 10.0.0.1:6443: i/o timeout - API server is overloaded or dead (this one always means bad news)

Step 2: Check If Your Nodes Are Still Alive

kubectl get nodes -o wide

If you see a bunch of NotReady status, don't panic yet. I've seen nodes show as NotReady for stupid reasons like:

• Network hiccup that lasted 30 seconds
• Node ran out of disk space because someone left debug logs running
• Cloud provider decided to restart the VM without telling anyone

Check the node conditions to understand what's actually broken. The kubelet logs usually have the real story. Pro tip: Take a screenshot of the node status. You'll forget the exact error when you're stressed and your manager asks what happened.

Step 3: What's Actually Running in kube-system?

kubectl get pods -n kube-system --sort-by=.status.phase

This shows you what control plane components are actually alive. Key things to look for:

• kube-apiserver pods stuck in Pending = you're totally fucked
• etcd pods in CrashLoopBackOff = data corruption, probably need to restore from backup
• kube-controller-manager failing = workloads won't get scheduled

Had our EKS cluster go down once because AWS rotated some cert we didn't know about. Spent forever troubleshooting the wrong thing because their error messages are garbage. Check the EKS troubleshooting guide if you're on AWS.

For Managed Clusters: Check Your Cloud Provider First

If you're running EKS, GKE, or AKS, check the cloud provider console before diving deep. Half the time it's:

• Planned maintenance they forgot to announce
• Your account hit a quota limit
• Their control plane is having issues (happens more than they admit)

Had our EKS cluster mysteriously die during a product demo once. Spent 30 minutes debugging our apps before checking the AWS console - they'd deprecated the control plane version we were using and auto-upgraded us mid-demo. Thanks, AWS.

Don't feel bad about checking this first - I've debugged "mysterious" cluster issues that were just AWS having a bad day. Check GKE status and Azure status pages too.

Reality Check: How Long Will This Take?

Based on actual experience, not textbook estimates:

Quick Fixes (15-30 minutes on 3-node cluster, 45+ minutes on 20+ nodes):

• Config mistakes you can fix with kubectl
• Restarting stuck pods
• Certificate renewals (unless you hit the cert rotation bug, then it's 2+ hours)

Medium Pain (1-3 hours on small clusters, 4-6 hours on large ones):

• Node failures requiring replacement (AWS takes 20-30 minutes just to provision new nodes)
• etcd issues that aren't corruption (compaction alone took 45 minutes on our 500GB etcd last time)
• Network policy fuckups (CNI restarts cascade across all nodes)

You're Fucked (4+ hours minimum, potentially days):

• etcd corruption without recent backups (took us 12 hours to rebuild everything from manifests)
• Multiple node failures in a small cluster (lost quorum = start over from scratch)
• Control plane components completely missing (happened during our k8s upgrade disaster)

Pro tip: If you can't get basic kubectl commands working in 15 minutes, something is fundamentally broken and you need to escalate. Check the official troubleshooting docs and don't be afraid to call your cloud provider support - that's what you're paying for.

Control plane is down? Here's the shit that actually works. etcd corrupted? Time to find out if you have backups or if you're about to have a very long day.

Is the API Server Actually Dead?

Before you panic, check if the API server is just having a moment:

## Check if it's running at all
sudo systemctl status kube-apiserver
sudo journalctl -u kube-apiserver --since \"10 minutes ago\" -f

Common ways the API server dies:

• Certificate expired - Check /etc/kubernetes/pki/ for expired certs. This happens every year and we always forget. Certificate management is a pain.
• etcd can't be reached - If etcd is down, API server is useless. Fix etcd first.
• Out of memory - Check dmesg for OOM killer messages. API server got too hungry. Resource limits help prevent this.
• Bad config changes - Someone edited /etc/kubernetes/manifests/kube-apiserver.yaml and broke it. Static pod manifests are fragile.

Real error you'll see: connection refused usually means the process isn't running. timeout means it's running but fucked. Check the API server troubleshooting guide for more details.

etcd Problems (AKA You're Probably Fucked)

etcd is where Kubernetes keeps all its shit. When etcd breaks, everything breaks. And etcd breaks in creative ways. Don't try to fix etcd corruption without a backup - I learned this the expensive way.

Quick health check:

etcdctl endpoint health --cluster
etcdctl endpoint status --cluster -w table

What you're looking for:

• healthy = good
• connection refused = etcd process is dead
• context deadline exceeded = etcd is alive but not responding (usually disk I/O issues)
• database space exceeded = etcd filled up its disk

Check the etcd disaster recovery docs and the etcd maintenance guide before you panic. The etcd performance benchmarks will help you understand if your storage is the problem.

Oh Shit, You Have Backups (Right?)

If you have etcd backups (please tell me you do), here's how to restore them:

## Stop everything first
sudo systemctl stop kubelet
sudo systemctl stop etcd

## Backup your corrupted data (just in case)
sudo cp -r /var/lib/etcd /var/lib/etcd-broken-$(date +%Y%m%d)

## Restore from backup
ETCDCTL_API=3 etcdctl snapshot restore /path/to/your/backup.db \
  --data-dir /var/lib/etcd-new \
  --name node1 \
  --initial-cluster node1=10.0.1.10:2380 \
  --initial-advertise-peer-urls 10.0.1.10:2380

Critical: Don't fuck up the --initial-cluster URLs. Use the actual IP addresses from your etcd config.

Time estimate: If everything goes right, took us like 25 minutes last time. If you need to figure out the cluster configuration, could be 2-3 hours. If you don't have backups, you're rebuilding the cluster.

Pro tip from our worst incident: Don't try to restore etcd during business hours. We thought it'd take 20 minutes. Took 3 hours and the entire engineering team was watching us like hawks.

You Don't Have Backups, Do You?

No judgment. We've all been there. Here are your options:

Option 1: Try to recover etcd data

## Sometimes etcd just needs a kick
sudo systemctl stop etcd
sudo etcd --data-dir /var/lib/etcd --force-new-cluster

This works maybe 30% of the time. Worth trying before giving up.

Option 2: Rebuild the cluster
If etcd is truly fucked and you don't have backups:

1. Accept that all your ConfigMaps, Secrets, and custom resources are gone
2. Rebuild the control plane from scratch
3. Redeploy everything from your manifests (you do have those in git, right?)
4. Start taking etcd backups

Multi-Node etcd: Extra Fun

If you have a 3-node etcd cluster and only one node is corrupted, you might get lucky:

## Remove the broken member
etcdctl member remove <member-id>

## Add a new member
etcdctl member add node3 --peer-urls=https://10.0.1.12:2380

But if 2 out of 3 nodes are fucked, you need to restore from backup on all nodes simultaneously. This is a pain in the ass and usually takes 3+ attempts.

Scheduler and Controller Manager: The Easy Ones

Once API server and etcd are happy, check these:

kubectl get pods -n kube-system | grep -E \"(scheduler|controller)\"

If they're in CrashLoopBackOff:

• Check the logs: kubectl logs -n kube-system <pod-name>
• Usually it's a config issue or they can't connect to the API server
• Restart them: kubectl delete pod -n kube-system <pod-name>

Did It Actually Work?

Don't declare victory until you've verified:

## Can you create stuff?
kubectl run test --image=nginx --rm -it -- /bin/bash

## Are nodes ready?
kubectl get nodes

## Is DNS working?
kubectl run busybox --image=busybox:1.28 --rm -it -- nslookup kubernetes.default

If any of these fail, something is still broken.

Pro tip: Don't tell anyone it's fixed until you've verified workloads can actually be created and scheduled. I once declared an incident resolved only to find out 10 minutes later that DNS was completely broken. Use the Kubernetes debugging guide and verify with the health check procedures before celebrating.

Control plane is back? Great. Now comes the hard part: getting your apps running again without breaking more shit. I've brought clusters back from the dead 8 times in 3 years, and learned some painful lessons.

First Things First - Are Your Nodes Actually Working?

Before you start deploying shit, make sure your nodes aren't lying to you:

## See what's actually broken
kubectl get nodes -o wide

Nodes showing NotReady? SSH to one and check:

## Is kubelet running?
sudo systemctl status kubelet

## What's kubelet complaining about?
sudo journalctl -u kubelet --since \"30 minutes ago\" | tail -20

## Did we run out of disk space? (classic)
df -h

Common reasons nodes are fucked:

• Disk full - Usually /var/lib/docker or /var/log filled up
• Network issues - Can't talk to other nodes or control plane
• Container runtime died - Docker/containerd crashed and didn't restart
• Out of memory - Node got OOM killed

Quick fixes that usually work:

• Restart kubelet: sudo systemctl restart kubelet
• Clean up disk space: docker system prune -a (if using Docker)
• Reboot the node if you're desperate

Check the kubelet troubleshooting guide and node debugging docs for more details. The container runtime documentation explains what can go wrong with Docker/containerd.

Fix Things in the Right Order (Don't Be a Hero)

Order matters. Don't try to bring everything back at once - I learned this the expensive way during the Memorial Day clusterfuck of 2022 when I restarted everything simultaneously and created a resource stampede that took us down for another 2 hours.

## See what's broken
kubectl get pods --all-namespaces | grep -v Running

Here's the order that actually works:

Start with system stuff in kube-system namespace. If this is fucked, nothing else matters:

kubectl rollout restart deployment coredns -n kube-system
kubectl get pods -n kube-system

DNS must work next - everything depends on this. Test it:

## This should work, or you're still fucked
kubectl run busybox --image=busybox:1.28 --rm -it -- nslookup kubernetes.default

Then databases. Apps can't work without data, and database recovery takes the longest.

Your actual applications come last. Don't be a hero and try to bring everything up at once.

Wait for each stage to be stable before moving to the next. I've seen people restart everything at once and create a resource stampede that brought the cluster down again. Don't be that person.

Follow the deployment troubleshooting flowchart and check the DNS debugging guide if things get weird. The resource quotas documentation explains how to prevent resource stampedes.

Check Your Storage (This Is Where Things Get Scary)

Storage problems during outages can mean data loss. Check this carefully:

## Are PVCs still bound?
kubectl get pvc --all-namespaces

If you see Pending PVCs, your storage is fucked:

## What's the storage class situation?
kubectl get storageclass
kubectl describe storageclass gp2  # or whatever your default is

Test if storage actually works:

## Create a test PVC
kubectl create -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes: [\"ReadWriteOnce\"]
  resources:
    requests:
      storage: 1Gi
EOF

## Did it bind?
kubectl get pvc test-pvc

If this fails, your databases are probably fucked and you need to restore from backup.

For databases: Don't assume the data is fine just because the pod started. Connect to the DB and verify the data is actually there and consistent. Check the persistent volume troubleshooting guide and storage class documentation for storage-specific issues.

Test Everything (Don't Trust, Verify)

Don't declare victory until you've actually tested that things work. I once told everyone we were good, then DNS shit the bed 10 minutes later during the all-hands meeting. Looked like a complete idiot in front of the CEO.

## Can you create new pods?
kubectl run test-pod --image=nginx --rm -it -- /bin/bash

## Is networking working?
kubectl exec test-pod -- curl kubernetes.default.svc.cluster.local

## Can apps talk to each other?
kubectl exec test-pod -- nslookup your-database-service

## Are your ingresses working?
curl -H \"Host: yourapp.example.com\" 127.0.0.1:8080

Monitor resource usage for the next hour:

## Watch for resource pressure
kubectl top nodes
kubectl top pods --all-namespaces

## Look for warning events
kubectl get events --all-namespaces --field-selector type=Warning

Things to watch for after recovery:

• CPU spikes above 80% for more than 10 minutes (Redis took 30 minutes to rebuild caches last time)
• Disk I/O wait time over 15% (PostgreSQL spent 2 hours catching up after our last incident)
• Network connections stuck in TIME_WAIT (saw 40k stuck connections during recovery)
• Apps throwing connection timeouts (database connection pools take forever to reset)

Cloud Provider Reality Check:

• AWS EKS node groups take forever to drain (seriously, like 20+ minutes). Had one take 45 minutes during our Black Friday recovery
• GKE node pools fail to delete for mysterious reasons about 30% of the time. "Node pool is in use" errors even when nothing's running
• Azure AKS throws weird RBAC permission errors during recovery. "User does not have permission to view workloads" even with cluster-admin role

Write Down What Happened (Before You Forget)

Trust me, you'll forget the details in a week. Write it down while it's fresh. I've been in too many postmortems where we couldn't remember what we actually did because nobody wrote it down.

What to document:

• When shit started breaking (with timestamps)
• What you tried that didn't work (so you don't waste time next time)
• What actually fixed it (the real solution, not the 10 things you tried before)
• How long each step took (for realistic time estimates)

Here's the actual postmortem from when our cluster died during a demo to investors in Q3 2023:

## Kubernetes Cluster Outage - September 14, 2023

### What Broke:
- Cluster went down at 2:17 PM during investor demo, everyone watching
- All nodes showed NotReady, CEO asked \"is this normal?\"
- Demo app returned 503s, investors not impressed

### Root Cause:
- etcd disk filled up during the demo (worst possible timing)
- We never monitored etcd disk usage like idiots
- etcd compaction wasn't configured properly

### Timeline (What Actually Happened):
1. 2:17 PM - Demo breaks, panic begins
2. 2:17-2:27 PM - Tried restarting random shit, made it worse
3. 2:27 PM - Finally checked disk space, etcd partition full
4. 2:27-2:52 PM - Cleaned up etcd, took 25 minutes
5. 2:52-3:22 PM - Waited for nodes to rejoin (30 minutes)
6. 3:22-5:30 PM - Spent 2+ hours fixing apps that broke during outage
7. 5:30 PM - Finally declared victory

### What We Learned:
- Monitor etcd disk usage or you'll get fucked like we did
- Set up automated etcd compaction before you need it
- Maybe test backups more than never
- Don't schedule demos during maintenance windows

Key point: Focus on what you learned, not who's to blame. We've all fucked up. Use the post-incident review template and check out Google's SRE postmortem culture for guidance on blameless postmortems.