Project Calico - Production-Ready CNI Technical Reference

Overview

What it is: Container Network Interface (CNI) plugin for Kubernetes networking
Production scale: 8+ million nodes worldwide
Core architecture: Pure L3 routing without overlay networking
Key differentiator: Actually debuggable when networking breaks

Configuration Options

Dataplane Selection

iptables Mode (Default - Recommended for Most)

• Use when: Starting new deployments, need debugging tools
• Performance: Standard IP routing performance
• Debugging: Full compatibility with tcpdump, wireshark, standard network tools
• Resource usage: ~100MB memory per node
• Failure modes: Visible through standard iptables commands
• Breaking point: Performance degrades with >50k concurrent connections per node

eBPF Mode (Performance-Focused)

• Use when: Hit performance walls with iptables mode
• Performance: 25% better throughput than iptables in high-connection scenarios
• Debugging: CRITICAL WARNING: Standard network debugging tools won't work
• Resource usage: 200MB+ memory per node
• Prerequisites: Engineers must understand eBPF debugging tools
• Breaking point: Monitoring systems depending on iptables packet counters will break

Windows Support

• Reality: Actually works, unlike other CNIs
• Features: Network policies function, service discovery reliable
• Limitation: Still Windows - expect standard Windows networking quirks

VPP Mode (Extreme Performance)

• Use when: Need maximum packet processing performance
• WARNING: Documentation sparse, debugging extremely difficult
• Time investment: Expect weeks of configuration troubleshooting
• Recommendation: Only for teams with dedicated VPP expertise

Network Routing Modes

BGP Mode (Best Performance)

• Advantages: No encapsulation overhead, line-rate packet forwarding
• Requirements: Network team must understand BGP concepts
• Scaling limitation: Requires route reflectors at 100+ nodes
• Cloud provider issues: GCP makes BGP unnecessarily complex
• Failure scenario: BGP misconfiguration can break inter-node communication completely

IP-in-IP Mode

• Use when: BGP not feasible in environment
• Performance cost: ~10% overhead from encapsulation
• Compatibility: Works in most cloud environments

VXLAN Mode

• Use when: IP-in-IP also blocked (rare scenarios)
• Performance cost: Higher overhead than IP-in-IP
• Troubleshooting: More complex packet analysis required

Critical Production Warnings

Scaling Breakpoints

• 50-100 nodes: Plan route reflector deployment
• 500+ nodes: Monitor etcd performance closely - becomes bottleneck before Calico
• 1000+ nodes: Felix memory usage scales with endpoint count
• Performance degradation: Policy evaluation time increases with policy complexity

Common Failure Modes

Felix Crashes

• Symptom: Networking dies completely
• Root cause: Usually OOM due to insufficient memory limits
• Prevention: Don't use default memory limits in production
• Debug command: kubectl logs -n kube-system -l k8s-app=calico-node

Network Policy Issues

• Reality: Most CNIs don't implement policies or implement them poorly
• Debugging sequence:
  1. calicoctl get workloadendpoints - verify Calico sees pods
  2. calicoctl get networkpolicies - confirm policies loaded
  3. iptables -L | grep cali - check actual firewall rules
• Critical feature: Policy preview prevents production breakage

Route Distribution Problems

• Symptom: Pods can't reach other nodes
• Debug commands:
  • ip route - check local routing table
  • BIRD BGP status for route advertisement issues
• Common cause: BGP misconfiguration in multi-node setups

Resource Requirements

Time Investment

• Initial setup: 15 minutes if experienced, 2 hours typical
• Learning curve: Moderate - BGP networking concepts helpful
• Debugging expertise: Standard networking tools work (unlike eBPF CNIs)

Infrastructure Requirements

• Memory: Minimum 100MB per node, scales with endpoint count
• Network: BGP support preferred for best performance
• etcd: Performance becomes bottleneck before Calico scales limit

Decision Criteria vs Alternatives

Calico vs Cilium

• Calico advantages: Debuggable with standard tools, lower resource usage
• Cilium advantages: Advanced L7 policies, eBPF-native features
• Choose Calico when: Need reliable debugging, standard networking teams
• Choose Cilium when: Need L7 policies, have eBPF expertise, CPU resources abundant

Calico vs Flannel

• Flannel limitation: No network policies - security audit failure
• Performance: Flannel VXLAN overhead vs Calico native routing
• Migration: Complete cluster rebuild required - no live migration possible

Calico vs Cloud Provider CNIs

• AWS VPC CNI: Calico provides network policies AWS CNI lacks
• GCP native: Calico works across cloud providers
• Vendor lock-in: Calico enables multi-cloud deployments

Implementation Requirements

Prerequisites

• Kubernetes versions: 1.27-1.31 supported (check compatibility matrix)
• Network knowledge: BGP understanding beneficial but not required
• Team skills: Standard networking troubleshooting sufficient

Installation Gotchas

• Test clusters first: Don't start with production
• Windows nodes: Full support available unlike other CNIs
• Air-gapped environments: Mirror all container images to internal registry

Monitoring Setup

• Flow logs: Now available in open source (Calico 3.30)
• Web UI: Whisker console better than parsing JSON logs
• Key metrics: Felix restart frequency, policy evaluation time
• Alert on: Felix OOMKilled events, BGP peer down events

Operational Intelligence

When Networking Breaks (3AM Debugging)

1. Pod connectivity issues:
   • Check kubectl get pods -o wide
   • Verify with calicoctl get workloadendpoints
   • Examine ip route for missing routes
2. Policy troubleshooting:
   • Use policy preview feature before enforcement
   • Start with deny-all policies, add allow rules
   • Avoid complex overlapping policies - performance killer

Migration Reality

• From other CNIs: Complete cluster rebuild required
• Zero-downtime claims: Marketing fiction - plan for downtime
• Strategies: Blue-green cluster replacement or rolling node replacement

Commercial vs Open Source

• Open source covers: 90% of use cases including flow logs (3.30+)
• Pay for commercial when: Need compliance reporting, multi-cluster visibility, support SLA
• Cost consideration: Support contract vs internal expertise investment

Production Hardening

• Security: Start with deny-all network policies
• Encryption: WireGuard integration available with 5-10% performance cost
• High availability: Felix failure affects only local node networking
• Backup strategy: Configuration stored in etcd with cluster backups

Critical Success Factors

Team Readiness

• Required skills: Basic Kubernetes networking, iptables understanding
• Helpful knowledge: BGP concepts for advanced deployments
• Avoid if: Team only familiar with overlay networking concepts

Environmental Fit

• Ideal for: Multi-cloud, hybrid cloud, bare metal deployments
• Cloud specific: Works everywhere, not tied to single provider
• Scale requirements: Proven at 1000+ node scale in production

Operational Maturity

• Day 1: Installation straightforward with standard manifests
• Day 2: Debugging uses standard networking tools
• Long term: Scales predictably, failure modes well understood

Bottom Line Assessment

Choose Calico when: Need reliable, debuggable networking that works everywhere
Avoid Calico when: Team lacks basic networking knowledge or needs only simple pod-to-pod communication
Success predictor: 8+ million production nodes don't lie - it works when configured correctly

The operational reality: Calico dominates because when networking breaks at 3AM, you can actually debug and fix it instead of restarting everything and hoping for the best.