Kubernetes Networking Breaks. Here's How to Fix It.

Kubernetes networking fails in predictable ways. I've been debugging this shit since Kubernetes 1.11, back when nobody knew what a CNI was and we all just hoped Flannel wouldn't randomly die. Here's what actually breaks and how to fix it without losing your mind.

CNI Plugin Failures

The CNI plugin handles all pod networking. When it breaks, you get these fun symptoms:

• Nodes stuck in NotReady with "CNI plugin not initialized"
• Pods stuck in Pending forever
• Random connectivity drops that make you question reality
• failed to create pod sandbox errors in kubelet logs

What's actually wrong:

CIDR conflicts - Your pod network overlaps with node or service networks. I've seen this kill entire clusters during weekend deployments.

## See what networks you're using
kubectl cluster-info dump | grep cidr
kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'

Version mismatches - Your CNI plugin doesn't support your Kubernetes version. Kubernetes 1.25 changed the default CNI timeout from 10s to 30s, which masks real connection issues. Don't use Flannel 0.15.1 - it corrupts routing tables on node restart. Calico 3.20+ requires Kubernetes 1.19+ but the error messages just say "plugin failed" without mentioning version conflicts.

IP exhaustion - Someone configured a /28 subnet for 100 pods. Math doesn't work.

## Check what IPs you have left
kubectl describe node NODE_NAME | grep PodCIDR
## For Calico users
calicoctl get ippool -o wide

The fix? Expand your CIDR or reduce your pod density. I learned this the hard way during Black Friday 2021 when AWS was having one of their "everything is fine" us-east-1 outages and we were frantically trying to failover to us-west-2, only to discover our pod CIDR was a /24 and we needed to scale to 500 pods. Took down checkout for 3 hours.

DNS Resolution Failures

CoreDNS is supposed to handle DNS but breaks constantly:

• Service resolution works sporadically (like 70% of the time)
• nslookup kubernetes.default returns SERVFAIL
• Apps can't find services that definitely exist
• DNS works from some pods but not others

What's actually wrong:

CoreDNS resource starvation - Default resource limits are garbage. 100m CPU isn't enough for any real load. Resource limits that look reasonable will throttle DNS under load.

## Check if CoreDNS is being throttled
kubectl top pods -n kube-system | grep coredns
kubectl describe deployment coredns -n kube-system | grep resources

Quick fix: Bump CoreDNS resources to 500m CPU and 512Mi memory. The default limits are a joke - whoever thought 100m CPU would handle a production cluster was clearly not running real workloads.

DNS config inconsistency - Different nodes have different DNS settings after upgrades. Kubelet configurations get out of sync.

## Check if DNS configs match across nodes
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide
kubectl describe configmap coredns -n kube-system

Network policies breaking DNS - Someone applied network policies without understanding they need to allow DNS traffic. Pods can ping by IP but can't resolve names.

Service Routing Issues

Kubernetes Services are load balancers that route traffic to pods. They break constantly.

You'll see:

• kubectl get svc shows endpoints exist
• Direct pod access works: curl pod-ip:8080
• Service access fails: curl service-name:8080
• Load balancer says "no healthy targets"

What's broken:

Endpoint lag - The endpoint controller is slow updating when pods start/stop. Your service sends traffic to dead pods.

## Check if endpoints match reality
kubectl get endpoints your-service -o yaml
kubectl get pods -l app=your-app -o wide

Mixed kube-proxy modes - Some nodes use iptables, others use ipvs. kube-proxy configuration is inconsistent after upgrades.

External Access Problems

Ingress controllers handle external traffic. They fail in spectacular ways:

• Ingress never gets an external IP
• 502 errors for services that work internally
• SSL cert issues causing browser warnings
• Traffic routes to wrong backends

Common failures:

Cloud LB integration broken - AWS ALB can't create load balancers due to IAM issues. GCP quota limits hit.

## Check ingress status
kubectl get ingress -A
kubectl describe ingress your-ingress
kubectl logs -n ingress-nginx deployment/ingress-nginx-controller

Cert-manager failures - cert-manager can't renew certs because DNS challenges fail or HTTP challenges are blocked.

Network Policy Hell

Network policies break legitimate traffic more than they block attacks. Default deny-all policies applied without understanding what needs to communicate.

## See what policies are blocking you
kubectl get networkpolicies -A
kubectl get namespaces --show-labels

The Debug Process That Actually Works

When everything's broken, use this order:

1. Check CNI status

kubectl get nodes -o wide
kubectl describe nodes | grep Ready

2. Test basic connectivity

kubectl run test --image=busybox --rm -it -- ping 8.8.8.8

3. Verify DNS

kubectl exec test -- nslookup kubernetes.default

4. Test service routing

kubectl exec test -- curl service-name:8080

5. Check external access

kubectl exec test -- curl your-domain.com

Start here and work through systematically. Most networking issues are CNI problems, DNS throttling, or network policies blocking legitimate traffic.

Sometimes kubectl get pods doesn't tell you why everything's broken. I've spent countless nights debugging this crap across different environments - on-prem bare metal where you don't have cloud provider magic, GKE where Google decides to "help" by changing your CNI config, and AWS where EKS updates randomly break your custom CNI settings.

CNI-Specific Debugging - Every Plugin Breaks Differently

I've debugged every major CNI plugin, and they all break in their own special ways. Here's how to debug each one when the basic stuff doesn't work.

Calico - When BGP Goes to Hell

Calico loves its BGP routing and complex architecture. When it breaks, you need calicoctl to figure out what's actually happening.

## Check Calico system status
calicoctl node status
calicoctl get nodes -o wide

## Debug IP allocation and routing
calicoctl get ippool -o wide
calicoctl get wep --all-namespaces | grep your-pod-name

## Verify BGP peering (if using BGP mode)
calicoctl node status
sudo calicoctl node diags

Calico-specific failure modes:

• BGP Session Failures: Nodes can't establish BGP peering, causing routing problems between nodes
• IP Pool Exhaustion: IPAM has run out of available IPs in the configured ranges
• Felix Agent Crashes: The Calico agent on nodes crashes under high policy evaluation load

Real war story: Had some weird issue where pods were getting duplicate IPs during high load back in Calico 3.18. Took me 6 hours to figure out - turns out there was a race condition in Calico's IPAM when using Kubernetes 1.20 with aggressive scaling policies. Calico would assign the same IP to multiple pods during rapid scale-up events. Fixed by downgrading to Calico 3.17 temporarily, then upgrading to 3.19 which actually fixed the race condition. Also had to tune the block allocation size from 64 to 26 because our nodes were small.

Cilium - eBPF Debugging Hell

Cilium is powerful but I fucking hate debugging eBPF issues - it feels like reading kernel assembly with a hangover. Full disclosure: I'm biased against Cilium's complexity, but it's the only thing that handles our scale without melting under 50k+ pods.

## Check Cilium agent status on nodes
kubectl exec -n kube-system ds/cilium -- cilium status
kubectl exec -n kube-system cilium-pod-name -- cilium endpoint list

## Debug eBPF program loading and packet processing
kubectl exec -n kube-system cilium-pod-name -- cilium bpf lb list
kubectl exec -n kube-system cilium-pod-name -- cilium monitor --type=drop

## Verify service connectivity
kubectl exec -n kube-system cilium-pod-name -- cilium service list

Cilium-specific debugging tools:

• Traffic monitoring: cilium monitor shows real-time packet flows and drops
• Policy tracing: cilium policy trace simulates policy evaluation for specific traffic
• eBPF inspection: cilium bpf commands inspect loaded eBPF programs and maps

Flannel - Simple Until It Isn't

Flannel is supposed to be the simple CNI. Works great until the VXLAN tunnels decide to shit themselves. If you're running on GKE, the default pod CIDR conflicts with most corporate VPNs - learned this the hard way when our entire engineering team couldn't VPN in after a cluster upgrade.

## Check Flannel pod status and configuration
kubectl get pods -n kube-flannel -o wide
kubectl logs -n kube-flannel ds/kube-flannel-ds

## Verify VXLAN tunnel interfaces on nodes
ip link show flannel.1
ip route show | grep flannel

## Check subnet allocation
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"	"}{.spec.podCIDR}{"
"}{end}'

Flannel troubleshooting scenarios:

• VXLAN MTU issues: Packets larger than the tunnel MTU get fragmented or dropped
• Routing table inconsistencies: Nodes have different views of pod subnet assignments
• Backend configuration mismatches: Mixing host-gw and vxlan backends causes routing confusion

Service Mesh Debugging - Adding More Ways to Break

Service meshes like Istio and Linkerd promise to solve all your networking problems. They mostly just add new ways to break.

Istio - When Your Sidecar is Fucked

Istio loves its Envoy sidecars. When they break, everything breaks in mysterious ways.

## Check Envoy sidecar configuration and status
kubectl exec your-pod -c istio-proxy -- pilot-agent status
kubectl exec your-pod -c istio-proxy -- curl localhost:15000/config_dump

## Debug traffic routing and load balancing
kubectl exec your-pod -c istio-proxy -- curl localhost:15000/clusters
kubectl exec your-pod -c istio-proxy -- curl localhost:15000/stats | grep your-service

## Verify mutual TLS configuration
istioctl authn tls-check your-pod.your-namespace.svc.cluster.local
istioctl proxy-config cluster your-pod.your-namespace

Common Istio networking problems:

• Sidecar injection failures: Pods start without Envoy sidecars due to namespace labeling issues
• mTLS authentication failures: Automatic mTLS negotiation fails between services
• Traffic policy conflicts: Multiple VirtualServices or DestinationRules create conflicting routing rules

Linkerd Network Debugging

Linkerd provides simpler service mesh functionality with built-in observability tools.

## Check Linkerd proxy status
linkerd check
linkerd stat deploy/your-deployment

## Debug traffic between services
linkerd tap deploy/your-deployment
linkerd edges deployments

## Verify proxy configuration
kubectl exec your-pod -c linkerd-proxy -- curl localhost:4191/ready

When Your App is Slow and You Don't Know Why

Your app is slow and everyone's blaming the network. Here's how to prove it's not your fault (or figure out that it actually is).

Network Latency - Measuring the Pain

Is the network slow or is your code garbage? Let's find out.

## Test baseline network latency between nodes
kubectl run network-test --image=nicolaka/netshoot --rm -it -- sh
## Inside pod: ping node-ip
## Inside pod: iperf3 -c other-pod-ip

## Measure service discovery latency
time kubectl exec your-pod -- nslookup your-service
kubectl exec your-pod -- dig your-service.your-namespace.svc.cluster.local

Bandwidth and Throughput Testing

Identifying network bandwidth limitations helps distinguish between network capacity issues and application bottlenecks.

## Test inter-pod bandwidth
kubectl run iperf-server --image=networkstatic/iperf3 -- iperf3 -s
kubectl run iperf-client --image=networkstatic/iperf3 --rm -it -- iperf3 -c iperf-server-ip

## Test node-to-node network performance
kubectl debug node/node1 -it --image=nicolaka/netshoot
## From debug container: iperf3 -c node2-ip

Connection Pool and Circuit Breaker Analysis

Modern applications use connection pooling and circuit breakers that can mask network problems or create false positives.

## Check application connection metrics
kubectl exec your-pod -- curl localhost:8080/metrics | grep -E \"(connection|circuit)\"

## For applications using Envoy (Istio):
kubectl exec your-pod -c istio-proxy -- curl localhost:15000/stats | grep -E \"(upstream|circuit_breaker)\"

## Monitor connection states
kubectl exec your-pod -- netstat -an | grep :8080

Container Network Interface (CNI) Performance Tuning

CNI configuration significantly impacts network performance, and tuning these settings can resolve performance bottlenecks.

CNI Plugin Performance Configuration

Each CNI plugin has performance-related configuration options that affect throughput and latency.

Calico performance tuning:

## Example Calico configuration for high-performance networking
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    bgp: Enabled
    mtu: 1500
    nodeAddressAutodetectionV4:
      firstFound: true
  flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/

Cilium performance optimization:

## Cilium ConfigMap for performance tuning
apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
  enable-bpf-masquerade: \"true\"
  enable-ip-masq-agent: \"false\"
  tunnel: \"disabled\"  # Use native routing for better performance
  auto-direct-node-routes: \"true\"

Node-Level Network Optimization

Operating system network configurations impact CNI plugin performance and should be optimized for high-throughput workloads.

## Check current network buffer settings
sysctl net.core.rmem_max
sysctl net.core.wmem_max

## Optimize for high-throughput networking
echo 'net.core.rmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.core.wmem_max = 134217728' >> /etc/sysctl.conf
echo 'net.ipv4.tcp_rmem = 4096 87380 134217728' >> /etc/sysctl.conf

## Apply network optimizations
sysctl -p

Multi-Cluster Network Debugging

Organizations running multiple Kubernetes clusters face additional networking challenges when services need to communicate across cluster boundaries.

Cross-Cluster Service Discovery

Debugging service discovery across clusters requires understanding how different multi-cluster solutions handle DNS and service registration.

## For Submariner multi-cluster networking
subctl show networks
subctl show connections
subctl diagnose all

## For Istio multi-cluster setup
istioctl proxy-config cluster your-pod --fqdn=your-service.your-namespace.cluster2.local

Network Policy in Multi-Cluster Environments

Network policies become more complex in multi-cluster setups, where services in one cluster need to communicate with services in another.

## Debug cross-cluster network policy enforcement
kubectl describe networkpolicy cross-cluster-policy
kubectl get services -A | grep multi-cluster

Multi-cluster nightmare: Had an issue in late 2022 where API calls worked fine during the day but died at night during batch processing. Spent 3 weeks debugging it because management wouldn't approve cluster downtime for troubleshooting. Turns out the network policies were written by someone who left the company - they were matching pod counts instead of actual service identity. When pods scaled up at night from 10 to 200, legitimate traffic got blocked. Fixed it at 3am on a Tuesday after finally convincing my manager to let me delete all network policies temporarily. Don't write policies when you're sleep deprived.

Look, most networking issues are either DNS being DNS, CIDR conflicts, or someone's network policy blocking legitimate traffic. Start with the basic stuff from the previous section before diving into this advanced diagnostic hell.

Network policies break legitimate traffic more than they block attacks. Whoever thought default-deny for everything was a good idea never worked production at 3am.

Network Policy Debugging - The Three Biggest Fuckups

After dealing with this shit for years, here are the mistakes that kill everything:

The \"Just Added One Policy\" Trap

Most people don't realize that adding ANY network policy to a namespace changes the default from "allow all" to "deny all" for the pods it selects. I've seen this break entire platforms.

## Check all network policies affecting a namespace
kubectl get networkpolicies -n your-namespace -o yaml

## Find policies that might be blocking your traffic
kubectl describe networkpolicy -n your-namespace your-policy

## Check which pods are selected by a policy
kubectl get pods -n your-namespace --show-labels
kubectl get pods -n your-namespace -l \"app=your-app\" --show-labels

The trap: Zero policies = everything works. Add one policy = only that specific traffic works, everything else dies.

War story: Back in August 2023, our security team wanted "defense in depth" so they deployed a single ingress policy on a Friday at 4:30pm to allow web traffic. By Monday morning, our frontend couldn't talk to the database and customer support was flooded with complaints. Turns out when you add ANY policy to pods, those pods default to deny-all for everything not explicitly allowed. Frontend could receive web requests but couldn't make database calls. Should've taken 5 minutes to fix, but spent 4 hours debugging because I kept assuming it was DNS again. The security guy who deployed it was on vacation in Cancun.

Policy Selector Debugging

Network policy selectors use label matching, and subtle label mismatches cause policies to select unexpected pods or miss their intended targets.

## Debug pod selector matching
kubectl get pods -n your-namespace -o jsonpath='{range .items[*]}{.metadata.name}{\"	\"}{.metadata.labels}{\"
\"}{end}'

## Check namespace selector behavior
kubectl get namespaces --show-labels
kubectl get networkpolicy your-policy -o jsonpath='{.spec.ingress[*].from[*].namespaceSelector}'

## Verify policy application with kubectl debug
kubectl run policy-test --image=busybox --rm -it -- sh
## Test connectivity from inside the policy-test pod

The Label Selector Clusterfuck

90% of network policy problems are just label selectors matching the wrong shit.

## See what labels your pods actually have
kubectl get pods --show-labels | grep your-app
## Compare with what your policy is selecting
kubectl describe networkpolicy your-policy | grep -A5 \"Pod Selector\"

DNS Becomes DNS't

Network policies break DNS more than anything else. Your pods can't reach CoreDNS in kube-system, so everything fails with nslookup: can't resolve.

The DNS Fix That Actually Works

Copy this. It allows DNS traffic to CoreDNS without opening everything up:

## Test if DNS is broken
kubectl run test-dns --image=busybox --rm -it -- nslookup kubernetes.default
## If that fails, your network policy is blocking DNS

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: your-namespace
spec:
  podSelector: {}  # Apply to all pods in namespace
  policyTypes:
  - Egress
  egress:
  # Allow DNS to CoreDNS
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    - podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  # Allow external DNS (adjust IP ranges as needed)
  - to: []
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

The Three Network Policy Mistakes That Ruin Your Day

Stop trying to be comprehensive. Here are the only 3 network policy problems you actually need to fix:

1. You Applied One Policy and Broke Everything Else

The default behavior flip from "allow all" to "deny all" catches everyone.

## See what policies are actually applied
kubectl get networkpolicies -A
## Check if any pods are selected but missing egress rules
kubectl describe networkpolicy -A | grep -A10 \"Pod Selector\"

2. DNS is Blocked

Your pods can't resolve service names because DNS traffic is blocked.

## Test DNS immediately  
kubectl run dns-test --image=busybox --rm -it -- nslookup kubernetes.default
## If it fails: \"server can't find kubernetes.default\"
## Add the DNS policy from above

3. Service Mesh Sidecar Traffic is Blocked

Istio and Linkerd proxies need specific ports allowed or everything breaks.

## Check for sidecar connection errors
kubectl logs your-pod -c istio-proxy | grep -E \"(connection refused|connection reset)\"
## Fix: Add ports 15001, 15006, and 15090 to your egress rules

Stop Overthinking Network Policies

Look, 90% of network policy problems are one of those three issues above. Before you dive into cloud security groups or complex policy evaluation, fix the basic shit first:

1. Check if you accidentally enabled deny-all mode
2. Make sure DNS works
3. Allow your service mesh ports

If those three things are working and you still have connectivity issues, THEN it might be something exotic like AWS security groups blocking your pod CIDR or conntrack table exhaustion. But 95% of the time, it's one of the three basic fuckups above.

Don't write network policies when you're sleep deprived. Don't apply them on Friday at 5pm. And always have a way to quickly delete all policies when everything breaks - which it will, probably during the demo to your CEO.