Container Network Interface (CNI) - AI-Optimized Technical Reference

Configuration

Production-Ready CNI Config

{
  "cniVersion": "1.1.0",
  "name": "mynet", 
  "type": "bridge",
  "bridge": "cni0",
  "ipam": {
    "type": "host-local",
    "subnet": "10.244.0.0/16"
  }
}

Configuration Location: /etc/cni/net.d/ (lowest numbered file wins)
Binary Location: /opt/cni/bin/ (must be executable)
Critical: Invalid JSON silently breaks everything

CNI Operations

• ADD: Create network for pod
• DEL: Cleanup when pod dies
• CHECK: Verify network working
• VERSION: Report plugin capabilities

Resource Requirements

Performance Benchmarks (P50 latency, 3-node cluster)

• AWS VPC CNI: 0.12ms (EKS only, fastest)
• Cilium: 0.15ms (requires kernel 4.9+, high RAM usage)
• Calico: 0.18ms (most stable, BGP knowledge required)
• Flannel: 0.22ms (most reliable, no security features)

Scaling Impact

• At 10,000 RPS: 0.1ms latency difference = significant performance impact
• Cilium: High RAM consumption (similar to Chrome browser)
• AWS VPC CNI: IP address exhaustion at scale

Critical Warnings

Production Failures

CNI Plugin Switching:

• Impact: Complete cluster destruction, 6+ hour downtime
• Cause: Fundamental networking reconfiguration required
• Solution: Plan CNI choice during initial setup only

Silent Failure Mode:

• Symptom: Pods schedule but cannot communicate
• Detection: Check /var/log/pods for CNI errors first
• Common Cause: Missing/non-executable CNI binary in /opt/cni/bin/

Kernel Update Breakage (Cilium):

• Impact: 3+ hour production outages
• Cause: eBPF compatibility breaks with kernel versions
• Prevention: Test Cilium version compatibility before kernel updates

AWS VPC CNI IP Exhaustion:

• Symptom: 25% of pods stuck in pending state
• Cause: Each pod consumes real VPC IP address
• Solution: Enable IP prefix delegation

Configuration Gotchas

File Precedence:

• /etc/cni/net.d/ uses numerical ordering
• 00-broken.conf will override working configuration
• Production clusters broken by accidental low-numbered files

Managed Service Limitations:

• EKS/GKE/AKS control CNI configuration
• Customization often impossible
• Blessing until custom requirements arise

Plugin Selection Matrix

Use Case	Recommended Plugin	Alternative	Avoid
Learning Kubernetes	Flannel	Calico	Cilium
Production on cloud	Cloud provider CNI	Calico	Custom solutions
On-premises + policies	Calico	Cilium	Flannel
High performance apps	Cilium (with expertise)	Calico	Flannel
Avoiding weekend debugging	Managed service	Calico	Weave Net (dead)

Implementation Reality

What Official Documentation Doesn't Tell You

Cilium:

• Markets "revolutionary eBPF" but requires PhD-level networking knowledge
• RAM consumption rivals desktop browsers
• Breaks on kernel updates without warning
• Performance benchmarks are marketing-driven

Calico:

• "Enterprise-grade" documentation assumes BGP expertise
• Actually stable but complex setup
• Works well once configured properly

Flannel:

• Actually simple as advertised
• Zero security features (complete vulnerability)
• Perfect for development, terrible for production

AWS VPC CNI:

• "Native integration" until IP addresses run out
• Fastest performance but cloud-locked
• Hidden costs from rapid IP consumption

Debugging Systematic Approach

1. Check /var/log/pods for CNI errors (first step always)
2. Verify CNI binary: Exists and executable in /opt/cni/bin/
3. Validate JSON: CNI config in /etc/cni/net.d/
4. kubelet logs: CNI errors sometimes only appear here
5. Network policies: Temporarily delete to test connectivity

Common Failure Root Causes

"Failed to setup CNI" errors:

• 90% cause: Missing or non-executable CNI binary
• 10% cause: Invalid JSON configuration

Random networking loss:

• CNI plugin crash (check plugin pod health)
• Configuration corruption
• Kernel incompatibility (Cilium-specific)

Pod-to-pod communication failure:

• Network policies blocking traffic
• CNI routing table corruption
• IP address conflicts

Breaking Points and Failure Modes

Resource Exhaustion Thresholds

• AWS VPC CNI: IP addresses per subnet
• Cilium: Available RAM (no specific threshold documented)
• All plugins: Node capacity for network interfaces

Migration Pain Points

• Zero-downtime CNI switching: Impossible
• Plugin compatibility: No cross-plugin migration path
• Configuration rollback: Requires complete cluster rebuild

Hidden Costs

Human Time Investment:

• Cilium: Requires kernel and eBPF expertise
• Calico: BGP routing knowledge essential
• Flannel: Minimal learning curve

Operational Overhead:

• Managed services: Reduced flexibility
• Self-managed: Weekend debugging sessions
• Custom configurations: Tribal knowledge requirements

Decision Criteria for Alternatives

Choose managed CNI when:

• Team lacks deep networking expertise
• Uptime requirements exceed 99.9%
• Cost of engineer time exceeds service cost

Choose Cilium when:

• Performance requirements critical
• Team has kernel expertise
• Memory resources abundant

Choose Calico when:

• Network policies required
• Stable, proven solution needed
• BGP expertise available

Choose Flannel when:

• Learning environment
• Simple requirements
• Security not required

Troubleshooting Decision Tree

Pod networking failure?
├── Check CNI plugin pod health
│   ├── Crashed → Restart CNI pods
│   └── Healthy → Check configuration
├── New pods can't get networking?
│   ├── CNI binary missing → Reinstall CNI
│   └── IP exhaustion → Check IPAM settings
└── Existing pods lose connectivity?
    ├── Recent kernel update → Check eBPF compatibility
    └── Configuration changed → Restore from backup

Operational Intelligence

Community and Support Quality

• Cilium: Active development, marketing-heavy documentation
• Calico: Enterprise focus, comprehensive but complex docs
• Flannel: Simple project, straightforward documentation
• AWS VPC CNI: Good AWS documentation, limited outside AWS

Worth It Despite Drawbacks

• Cilium: Performance gains justify complexity for high-throughput applications
• Calico: Stability worth the BGP learning curve for production
• Managed CNI: Cost justified by reduced operational burden

Common Misconceptions

• "CNI plugins are interchangeable" → Migration requires cluster rebuild
• "Flannel is production-ready" → Zero security features
• "Cilium benchmarks apply everywhere" → Marketing numbers vs. real-world performance
• "Network policies work the same across plugins" → Implementation varies significantly

This technical reference enables AI systems to understand CNI selection, implementation risks, and operational requirements without the human emotional context while preserving all actionable intelligence and decision-support information.