CNI Debugging Guide: AI-Optimized Reference

Critical Triage (3-Minute Emergency Response)

Immediate Assessment Commands

# Test pod scheduling capability - CRITICAL FIRST CHECK
kubectl run test-pod --image=nginx --rm -it -- /bin/bash

# If ContainerCreating stuck forever = cluster-wide CNI failure
# If successful = localized issue only

Emergency Status Check Sequence

1. CNI plugin status: kubectl get pods -n kube-system | grep -E "(cilium|calico|flannel)"
2. Node readiness: kubectl get nodes -o wide (look for NotReady)
3. Recent failures: kubectl get events --sort-by='.lastTimestamp' | grep -i error | tail -10

Time Criticality: If new pods cannot schedule, you have minutes before production impact escalates.

Common Failure Patterns and Root Causes

"failed to setup CNI" Error (95% occurrence rate)

Root Causes (in order of frequency):

1. Missing CNI binary - /opt/cni/bin/ empty after node updates
2. Corrupted CNI config - Invalid JSON in /etc/cni/net.d/
3. Permission issues - Config files unreadable by kubelet
4. Multiple conflicting configs - Priority conflicts between CNI files

Diagnosis Commands:

# Check CNI binary existence
kubectl debug node/worker-node-1 -it --image=alpine
ls -la /opt/cni/bin/

# Validate CNI configuration
ls -la /etc/cni/net.d/
cat /etc/cni/net.d/*.conf

Recovery Actions:

# Restart CNI DaemonSet (fixes 80% of cases)
kubectl rollout restart daemonset/calico-node -n kube-system
kubectl rollout restart daemonset/cilium -n kube-system
kubectl rollout restart daemonset/kube-flannel-ds -n kube-flannel

# Nuclear config fix
rm /etc/cni/net.d/10-broken.conf
kubectl delete pod -n kube-system -l k8s-app=your-cni-plugin

"No Route to Host" Network Failures

Diagnostic Workflow:

kubectl exec -it failing-pod -- /bin/bash
ip route show                    # Empty = CNI routing failure
ping 8.8.8.8                    # External connectivity test
nslookup kubernetes.default.svc.cluster.local  # Internal DNS test

Decision Tree:

• Empty routing table → CNI plugin crashed during setup
• External ping fails + DNS works → Egress/masquerading issue
• DNS fails → CoreDNS connectivity blocked (usually NetworkPolicy)

IP Address Exhaustion (Silent Killer Pattern)

Symptoms: Existing pods functional, new pods stuck in ContainerCreating
Detection Commands:

# Calico
kubectl exec -n kube-system calico-node-xxxxx -- calicoctl ipam show

# Cilium  
kubectl exec -n kube-system cilium-xxxxx -- cilium status --verbose

# AWS VPC CNI
kubectl describe configmap aws-node -n kube-system

Resolution Options (ranked by implementation difficulty):

1. Clean unused pods (immediate, temporary)
2. Enable IP prefix delegation (AWS only, requires restart)
3. Expand pod CIDR (requires cluster restart - high risk)

Plugin-Specific Debugging Intelligence

Calico BGP Failures

Critical Check: BGP peering status determines cluster connectivity

kubectl exec -n kube-system calico-node-xxxxx -it -- bash
calicoctl node status  # Must show "Established" for all peers

Common BGP Failure Modes:

• Idle/Connect states → Firewall blocking TCP 179
• Wrong AS numbers → Full-mesh configuration error
• IP-in-IP issues → Encapsulation mode mismatch

BGP Loop Recovery (nuclear option):

kubectl delete pods -n kube-system -l k8s-app=calico-node

Cilium eBPF Diagnostics

Essential Status Checks:

kubectl exec -n kube-system cilium-xxxxx -it -- bash
cilium status --verbose
cilium connectivity test     # Comprehensive connectivity validation
cilium bpf endpoint list     # eBPF program verification

Critical Failure Indicators:

• "BPF filesystem not mounted" → /sys/fs/bpf missing (kernel update issue)
• "Kubernetes APIs unavailable" → API server connectivity/RBAC failure

eBPF Recovery:

# Node-level fix for BPF filesystem
mount -t bpf bpf /sys/fs/bpf
echo "bpf /sys/fs/bpf bpf defaults 0 0" >> /etc/fstab

AWS VPC CNI Resource Limits

ENI/IP Exhaustion Detection:

kubectl describe configmap aws-node -n kube-system

Scaling Solutions:

1. Enable prefix delegation: kubectl set env daemonset aws-node -n kube-system ENABLE_PREFIX_DELEGATION=true
2. Use larger instance types (immediate capacity increase)
3. Implement pod density limits (preventive measure)

Hidden Gotcha: Security groups must allow pod CIDR range for cluster DNS (port 53)

Flannel VXLAN Overlay Issues

VXLAN Validation:

ip link show flannel.1          # Interface existence
bridge fdb show dev flannel.1   # Neighbor discovery
tcpdump -i flannel.1 -n icmp   # Traffic analysis

Subnet Conflict Detection:

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
# Check for duplicate CIDR assignments

Nuclear Recovery Procedures

Complete CNI Reset Script

#!/bin/bash
# CAUSES DOWNTIME - Use only when all else fails

# Delete CNI DaemonSets
kubectl delete ds -n kube-system -l k8s-app=cilium
kubectl delete ds -n kube-system -l k8s-app=calico-node
kubectl delete ds -n kube-flannel -l app=flannel

# Per-node cleanup (run on each node)
for node in $(kubectl get nodes -o name); do
    kubectl debug $node -it --image=alpine -- sh -c "
    rm -rf /etc/cni/net.d/*
    rm -rf /opt/cni/bin/*
    ip link delete cilium_host 2>/dev/null || true
    ip link delete cilium_net 2>/dev/null || true
    iptables -F -t nat
    iptables -F -t filter
    iptables -F -t mangle"
done

# Reinstall CNI (example for Cilium)
helm upgrade --install cilium cilium/cilium --namespace kube-system

Production Operational Intelligence

Critical Monitoring Requirements

1. Debug pods per namespace - Pre-deployed with network tools for emergency access
2. CNI resource limits - Prevent Cilium consuming 4GB+ RAM and crashing nodes
3. Previous CNI version availability - Rollback faster than debugging at 3AM
4. Policy validation in development - Test failure scenarios before production

Failure Prevention Patterns

• Monitor IP pool utilization before exhaustion occurs
• Test kernel updates against CNI compatibility matrices
• Validate NetworkPolicies with temporary deletion during emergencies
• Maintain RBAC permissions for CNI service accounts after cluster upgrades

Emergency Decision Matrix

Symptom	Time to Impact	Primary Action	Fallback
New pods cannot schedule	Minutes	Restart CNI DaemonSet	Nuclear CNI reset
Inter-pod communication failing	Immediate	Check BGP/VXLAN status	Policy cleanup
DNS resolution broken	Immediate	Delete NetworkPolicies	CoreDNS restart
IP exhaustion warnings	Hours	Enable prefix delegation	Scale instance types

Critical Resource Thresholds

• Calico IP blocks: Monitor fragmentation above 80% utilization
• AWS ENI limits: Track per-instance-type maximums
• Cilium memory usage: Alert above 2GB per node
• BGP session count: Scale considerations for large clusters

Common Misconfigurations with High Impact

1. Multiple CNI configs with conflicting priorities - Silent failures
2. Windows line endings in JSON configs - Parser failures
3. Wrong API server endpoints - Post-upgrade authentication failures
4. Security group rules missing pod CIDR - DNS resolution blocks

This operational intelligence represents field-tested knowledge from production CNI debugging scenarios, optimized for systematic troubleshooting and rapid problem resolution.