Project Calico - The CNI That Actually Works in Production

I've deployed Calico on everything from 3-node dev clusters to 500+ node production environments. Here's why it's the CNI that doesn't make you want to throw your laptop out the window.

The Problem with Other CNIs

Most CNI plugins are either too simple (Flannel has no network policies) or too clever for their own good (Cilium's eBPF everything approach breaks half your debugging tools). Weave died, Antrea is VMware-only, and don't get me started on kube-router.

Calico hits the sweet spot: it actually implements Kubernetes network policies, performs well, and when things break, you can actually debug them.

How Calico Actually Works

Pure IP Routing: No VXLAN tunnels, no overlay networking overhead. Your pods get real routable IP addresses. When a packet needs to go from node A to node B, it just... goes there. Revolutionary concept, apparently.

BGP Distribution: Uses BGP to tell other nodes about local routes. Yes, the same BGP that powers the internet. It works because it's been battle-tested for decades.

Multiple Dataplane Options: Start with iptables (it just works), upgrade to eBPF when you hit performance walls. Windows nodes? Supported. VPP for extreme performance? Available if you hate yourself enough to configure it.

What's New in Calico 3.30 (Released 2025)

The biggest improvement is that they moved a bunch of previously paid features into the open source version. About damn time:

• Flow logs without paying: The Goldmane API lets you actually see what's happening on your network
• Web UI for debugging: Calico Whisker web console beats parsing JSON logs at 3am
• Test policies before they break production: Policy preview lets you see what would be blocked before enabling enforcement
• Gateway API support: If you're into that sort of thing, works with Envoy Gateway and Kubernetes Gateway API

The Components That Matter

Felix: The agent that runs on every node and does the actual work. Programs iptables rules, manages network interfaces, handles route advertising. When Felix breaks, your networking dies.

BIRD: Handles BGP route distribution. Basically tells other nodes "hey, I have pods at these IPs, route traffic to me." Works in full mesh mode until your cluster gets big enough that you need route reflectors.

Typha: A caching proxy that sits between Felix and etcd. Required for clusters over ~100 nodes unless you want to watch etcd melt under load.

Why Choose Calico Over The Alternatives

It works everywhere: Unlike CNIs tied to specific cloud providers or distributions, Calico runs the same on bare metal, AWS, GCP, Azure, and your crappy on-prem cluster.

Debugging doesn't make you cry: When network policies aren't working, you can actually figure out why. `calicoctl get workloadendpoints` and `iptables -L` still work.

Performance that doesn't suck: Native IP routing means packets don't get wrapped in three layers of tunneling. Your latency stays reasonable and throughput doesn't tank.

The 8+ million nodes running Calico worldwide aren't there because of marketing - they're there because when you're woken up at 3am by networking issues, Calico is the CNI that actually helps you debug and fix problems instead of forcing you to restart everything and hope for the best. When you need Kubernetes networking that just works, Calico delivers.

Dataplane Options That Don't Suck

eBPF Mode (When You Need Speed)

The eBPF dataplane is genuinely fast - about 25% better throughput than iptables in high-connection scenarios. But here's the catch: when networking breaks, good luck debugging it. Your standard tooling won't help, and you'll be staring at eBPF bytecode at 3am.

Use eBPF when:

• You've hit performance walls with iptables
• You have engineers who understand eBPF debugging
• You're not running legacy monitoring that depends on iptables packet counters

iptables Mode (The Reliable Choice)

Default dataplane that just works. When pods can't reach services, you can actually debug it with tools you already know. Performance is fine for most workloads - if you're hitting limits with iptables, you probably have bigger problems.

Windows Support (Actually Works)

Unlike every other CNI, Calico's Windows support isn't an afterthought. Network policies work, service discovery works, and it doesn't randomly break when Windows updates.

VPP Mode (For Masochists)

Vector Packet Processing delivers insane performance if you can figure out how to configure it. Documentation is sparse, debugging is hell, and you'll spend more time fighting VPP than benefiting from the performance.

BGP Routing (The Good and Bad)

The Good: No VXLAN tunnels means no encapsulation overhead. Packets flow at line rate because they're just regular IP packets. BGP has been running the internet for decades, so it's battle-tested.

The Bad: Your network team needs to understand what's happening. Some cloud providers (looking at you, GCP) make BGP more complex than it needs to be. Route reflectors become necessary around 100+ nodes.

The Reality: BGP mode gives you the best performance but requires network knowledge. IP-in-IP mode works everywhere but adds ~10% overhead. VXLAN mode exists for environments where even IP-in-IP doesn't work (rare).

Network Policies That Actually Work

The Reality of Network Policies

Most CNIs either don't support network policies (Flannel) or implement them poorly. Calico's policies actually do what they say they'll do.

## This will block everything except what you explicitly allow
apiVersion: projectcalico.org/v3
kind: NetworkPolicy
metadata:
  name: deny-all-default
spec:
  selector: all()
  types:
  - Ingress
  - Egress

Pro tip: Start with deny-all policies and add allow rules. The opposite approach (allow-all then deny) is security theater.

Policy Debugging Hell (And How to Escape)

When network policies don't work as expected:

1. `calicoctl get workloadendpoints` - see what Calico thinks your pods are
2. `calicoctl get networkpolicies` - verify policies are actually loaded
3. `iptables -L | grep cali` - see the actual iptables rules

The policy preview feature is a godsend - you can test what would be blocked before enabling enforcement.

WireGuard Encryption

Works out of the box with minimal performance impact (~5-10%). Way easier than setting up service mesh encryption, and it encrypts node-to-node traffic that service mesh misses.

Enable it once your cluster is stable - don't add encryption complexity to initial deployments.

Observability That Doesn't Suck

Flow Logs (Finally Free)

Calico 3.30 moved flow logs from paid to free. About time. The Goldmane API gives you JSON logs of what's actually happening on your network, which beats guessing when debugging connectivity issues.

The Whisker web console is decent for filtering through logs - better than parsing gigabytes of JSON by hand.

Real Debugging Workflow

When networking is broken (and it will be):

1. Check if pods are up: kubectl get pods
2. Check if Calico sees them: calicoctl get workloadendpoints
3. Check policy evaluation: calicoctl get networkpolicies --show-labels
4. Look at flow logs for denied connections
5. Check iptables rules: iptables -L | grep cali

The flow logs finally let you see why connections are failing instead of just that they are.

The Production Reality

When Calico Scales (And When It Doesn't)

Calico works well up to 1000+ nodes. The largest deployments run millions of nodes across multiple clusters. But:

• Route reflectors become mandatory around 100+ nodes
• etcd performance becomes the bottleneck before Calico does
• Felix memory usage grows with the number of endpoints
• Policy evaluation time increases with policy complexity

Cloud Provider Gotchas

• AWS: VPC CNI exists, but Calico works fine and gives you network policies
• GCP: Alias IP mode can be tricky, but worth it for performance
• Azure: Works out of the box, no special considerations
• Bare metal: BGP peering with your network gear actually works

Operational Headaches

• Upgrades: Usually smooth, but test policy evaluation after minor version bumps
• Monitoring: Felix restarts are normal, but frequent restarts indicate configuration issues
• Resource limits: Don't be cheap with Felix memory limits - OOMKilled Felix breaks networking
• IPv6: Supported but doubles your debugging complexity

The reality is that once Calico is running correctly, it tends to stay running. The hard part is getting the initial configuration right, but that's where this guide comes in - we've walked through the production gotchas so you don't have to learn them at 3am when your cluster is down.

This is why Calico dominates the CNI landscape. It's not perfect, but it's predictable. When things break, you can actually fix them. When you need to scale, it scales. When you need security policies, they work. And when you're evaluating CNI options for your next cluster, remember: the best CNI is the one that doesn't wake you up at night.