Kubernetes Network Troubleshooting: AI-Optimized Knowledge Base

Critical Failure Scenarios and Consequences

CNI Plugin Failures - Cluster-Wide Impact

Symptoms:

• Nodes stuck in NotReady with "CNI plugin not initialized"
• Pods stuck in Pending forever
• Random connectivity drops causing service degradation
• failed to create pod sandbox errors in kubelet logs

Critical Consequence: Entire cluster becomes unusable, all new pod deployments fail

Root Causes with Business Impact:

• CIDR Conflicts: Pod network overlaps with node/service networks → Complete cluster failure during weekend deployments
• Version Mismatches: CNI plugin incompatible with Kubernetes version → Silent failures that manifest under load
• IP Exhaustion: Insufficient CIDR allocation → Service unavailability during traffic spikes

Real-World Failure Example: Black Friday 2021 - AWS us-east-1 outage led to failover attempt blocked by /24 pod CIDR supporting only 254 IPs when 500 pods needed. Checkout down 3 hours.

DNS Resolution Failures - Application-Level Breakdown

Symptoms:

• Service resolution works 70% of the time (intermittent failures)
• nslookup kubernetes.default returns SERVFAIL
• Apps can't find services that exist
• DNS works from some pods but not others

Critical Consequence: Microservices architecture becomes unreliable, cascading failures across service dependencies

Resource Starvation Impact: Default CoreDNS limits (100m CPU) cause DNS throttling under any real load → Application timeouts and user-facing errors

Configuration That Actually Works in Production

CoreDNS Resource Requirements

Default Settings That Fail:

• CPU: 100m (insufficient for production)
• Memory: 170Mi (causes OOM under load)

Production-Tested Configuration:

• CPU: 500m minimum (handles real traffic)
• Memory: 512Mi minimum (prevents OOM kills)

Implementation:

kubectl patch deployment coredns -n kube-system -p '{"spec":{"template":{"spec":{"containers":[{"name":"coredns","resources":{"limits":{"cpu":"500m","memory":"512Mi"}}}]}}}}'

Network Policy Configuration That Doesn't Break Everything

Critical Understanding: Adding ANY network policy to a namespace changes default from "allow all" to "deny all" for selected pods.

Essential DNS Policy (Must-Have):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: your-namespace
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    - podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

Decision-Support Information

CNI Plugin Comparison with Operational Reality

CNI Plugin	Complexity	Failure Rate	Debug Difficulty	Production Readiness
Flannel	Low	Medium	Easy	Good for simple setups
Calico	Medium	Low	Medium	Best for scale/policies
Cilium	High	Medium	Very Hard	Powerful but complex

Calico Trade-offs:

• Worth it despite: BGP complexity and debugging requirements
• Hidden cost: Requires network engineering expertise
• Breaking point: BGP session failures cause cross-node communication loss

Cilium Trade-offs:

• Worth it despite: eBPF debugging complexity requiring kernel knowledge
• Performance benefit: Only solution handling 50k+ pods without performance degradation
• Hidden cost: Requires deep Linux networking expertise

Service Mesh Decision Matrix

Istio Implementation Reality:

• Time investment: 3-6 months to operational maturity
• Expertise required: Deep Envoy and mTLS knowledge
• Common failure: Sidecar injection breaks 20% of deployments initially
• Performance impact: 10-15% latency increase, 200MB memory overhead per pod

Linkerd Comparison:

• Easier than: Istio configuration and debugging
• Harder than: Basic Kubernetes networking
• Sweet spot: Teams wanting service mesh benefits without Istio complexity

Critical Warnings and Operational Intelligence

What Official Documentation Doesn't Tell You

Kubernetes 1.25 Changes:

• Default CNI timeout increased from 10s to 30s
• Hidden impact: Masks real connection issues by making them appear successful
• Debugging implication: Timeouts that would fail fast now hang for 30s

Flannel Version-Specific Issues:

• Flannel 0.15.1: Corrupts routing tables on node restart (avoid completely)
• GKE Default CIDR: 10.0.0.0/14 conflicts with most corporate VPNs
• Production impact: Engineering team VPN access blocked after cluster upgrades

Network Policy Production Failures

Three Documented Production Outages:

1. August 2023 - "Defense in depth" deployment

   • Cause: Single ingress policy deployed Friday 4:30 PM
   • Impact: Frontend couldn't reach database by Monday
   • Root cause: Policy enabled deny-all mode for database connections
   • Resolution time: 4 hours (assumed DNS, debugged wrong layer)

2. Black Friday scaling failure

   • Cause: /24 CIDR allocated for 500-pod requirement
   • Impact: Checkout system down 3 hours during peak traffic
   • Prevention: CIDR planning based on worst-case scaling scenarios

3. Network policy label mismatch

   • Cause: Policy matched pod count instead of service identity
   • Impact: Night batch processing blocked when pods scaled 10→200
   • Hidden cost: 3 weeks debugging, management wouldn't approve downtime

Common Misconceptions That Cause Failures

"Zero Network Policies = Secure Default"

• Reality: Zero policies = everything allowed
• Trap: Adding one policy = deny-all for non-matching traffic
• Fix timing: Plan comprehensive policies, deploy atomically

"Default Resource Limits Are Production-Ready"

• Reality: CoreDNS 100m CPU fails under any real load
• Impact: DNS throttling appears as application bugs
• Fix: 5x default limits minimum for production

"CNI Plugins Are Interchangeable"

• Reality: Each has specific failure modes and debugging requirements
• Migration cost: Complete cluster rebuild often required
• Expertise transfer: Team knowledge doesn't transfer between CNIs

Diagnostic Procedures with Time Investment

Systematic Network Debugging (15-30 minutes)

Layer-by-layer diagnosis approach:

1. CNI Health Check (2 minutes)

   kubectl get nodes -o wide
   kubectl describe nodes | grep Ready
   

2. Basic Connectivity Test (3 minutes)

   kubectl run test --image=busybox --rm -it -- ping 8.8.8.8
   

3. DNS Verification (2 minutes)

   kubectl exec test -- nslookup kubernetes.default
   

4. Service Routing Test (3 minutes)

   kubectl exec test -- curl service-name:8080
   

5. External Access Verification (5 minutes)

   kubectl exec test -- curl your-domain.com
   

Time-saving rule: 90% of issues found in steps 1-3, don't skip to complex debugging

CNI-Specific Debugging Time Investment

Calico Issues (30-60 minutes):

• BGP Status Check: 5 minutes with calicoctl
• IP Allocation Debug: 10 minutes understanding IPAM
• Policy Troubleshooting: 45 minutes for complex scenarios

Cilium Issues (2-4 hours):

• eBPF Program Analysis: Requires kernel debugging skills
• Policy Tracing: Complex evaluation logic
• Performance Impact: Often requires cluster-level changes

Resource Requirements and Expertise Costs

Human Time Investment by Problem Type

Problem Category	Initial Diagnosis	Full Resolution	Expertise Required
DNS Throttling	5 minutes	15 minutes	Basic kubectl
Network Policy	10 minutes	2 hours	Label selector understanding
CNI Failures	30 minutes	4 hours	Network engineering
Service Mesh	1 hour	8 hours	Deep proxy knowledge

Skill Prerequisites Not in Documentation

Network Policy Debugging:

• Required: Deep understanding of label selectors and namespace behavior
• Time to competency: 2-3 production incidents
• Common gap: Developers don't understand Kubernetes networking defaults

CNI Troubleshooting:

• Required: Linux networking, routing tables, iptables
• Time to competency: 6 months production experience
• Common gap: Cloud engineers lack on-premises networking knowledge

Service Mesh Operations:

• Required: TLS, proxy configuration, observability tools
• Time to competency: 3-6 months dedicated focus
• Common gap: Application developers lack infrastructure knowledge

Breaking Points and Failure Modes

Scale-Related Network Failures

CNI Performance Limits:

• Flannel: 100-200 nodes before BGP instability
• Calico: 1000+ nodes with proper BGP configuration
• Cilium: 5000+ nodes but requires eBPF expertise

DNS Performance Breakdown:

• CoreDNS: Becomes bottleneck at 500+ QPS with default limits
• Symptom: Intermittent resolution failures under load
• Fix cost: Resource tuning (easy) vs DNS caching architecture (complex)

Network Policy Complexity Limits

Management Overhead:

• 10-20 policies: Manageable with documentation
• 50+ policies: Requires automation and testing
• 100+ policies: Policy conflicts become undebuggable

Real-world breaking point: Teams abandon network policies after 3rd production outage caused by policy interactions

Tools Effectiveness Matrix

Debugging Tool Selection by Problem Type

Tool	Basic Connectivity	DNS Issues	Policy Debug	CNI Problems	Time to Result
kubectl logs	Limited	Good	Poor	Poor	30 seconds
kubectl describe	Good	Limited	Good	Good	1 minute
netshoot pod	Excellent	Excellent	Good	Limited	2 minutes
calicoctl	Poor	Poor	Excellent	Excellent	5 minutes
tcpdump	Excellent	Good	Poor	Excellent	10 minutes

Cost-Benefit Analysis of Debugging Approaches

Quick Wins (5-15 minutes):

• kubectl logs and describe commands
• Basic connectivity tests with busybox
• Resource limit verification

Medium Investment (30-60 minutes):

• Network policy analysis
• CNI-specific tooling
• Service mesh configuration review

Deep Debugging (2+ hours):

• Packet capture analysis
• eBPF program inspection
• Multi-cluster networking issues

ROI Guidance: Start with quick wins, escalate only when basic approaches fail

Migration and Change Management

Version Upgrade Risks

Kubernetes Version Changes:

• 1.24→1.25: CNI timeout behavior change masks issues
• 1.25→1.26: Network policy evaluation order changes
• Impact: Silent failures appearing weeks after upgrade

CNI Plugin Migrations:

• Flannel→Calico: Requires complete cluster rebuild
• Calico→Cilium: IP pool migration complexity
• Time investment: 2-4 weeks planning, 1 week execution

Operational Maturity Stages

Stage 1 - Basic Operations (0-6 months):

• Can debug DNS and basic connectivity
• Understands service networking
• Avoids network policies

Stage 2 - Intermediate (6-18 months):

• Deploys simple network policies safely
• Debugs CNI-specific issues
• Handles routine networking problems

Stage 3 - Advanced (18+ months):

• Designs complex network architectures
• Debugs service mesh issues
• Handles multi-cluster networking

Acceleration factors: Production incidents provide 10x learning rate compared to lab environments

Emergency Response Procedures

Network Policy Emergency Recovery

Immediate Action (1 minute):

kubectl delete networkpolicies --all -n affected-namespace

Verification (2 minutes):

kubectl run test --image=busybox --rm -it -- ping service-ip

Root Cause Analysis (15 minutes):

• Review policy selectors and namespace labels
• Test policy application with temporary pods
• Document policy interactions for future prevention

CNI Failure Recovery

Emergency Pod Restart (2 minutes):

kubectl delete pods -n kube-system -l k8s-app=flannel
# Or for other CNIs:
kubectl rollout restart ds/calico-node -n calico-system

Node-Level Recovery (5 minutes):

# Check and restart kubelet if needed
systemctl status kubelet
systemctl restart kubelet

Full CNI Reinstall (30 minutes):

• Deploy CNI manifests
• Verify node network configuration
• Test pod-to-pod connectivity

Quality and Support Indicators

Community and Vendor Support Quality

Calico/Tigera:

• Documentation quality: Excellent technical depth
• Community response: 24-48 hours for complex issues
• Enterprise support: Available with SLA guarantees

Cilium/Isovalent:

• Documentation quality: Good but assumes advanced knowledge
• Community response: Variable, depends on complexity
• Enterprise support: Required for production deployments

Flannel:

• Documentation quality: Basic, often outdated
• Community response: Slow, limited maintainer availability
• Enterprise support: None available

Tool Reliability Assessment

Production-Ready Tools:

• netshoot: Consistently reliable across environments
• calicoctl: Stable API, good backward compatibility
• kubectl: Core functionality stable, extensions variable

Experimental/Risky Tools:

• cilium CLI: Rapid development, breaking changes
• Custom network tools: Environment-specific reliability
• Alpha networking features: Not production suitable

This knowledge base provides the operational intelligence needed for AI systems to make informed decisions about Kubernetes networking troubleshooting, including understanding failure modes, resource requirements, and the real-world costs of different approaches.