When Kubernetes Network Policies Break Everything (And How to Fix It)

Network policies are trash. They look fine until you apply one and suddenly nothing works.

Here's the thing nobody tells you: applying ANY network policy to a pod flips a switch. Before policies, everything can talk to everything. After policies, nothing can talk to anything unless you explicitly allow it. It's like going from an open house party to a bouncer-protected VIP club.

Deployed a "simple" policy to our frontend namespace once. Everything started returning 503s immediately. Frontend couldn't reach the API, API couldn't hit the database, monitoring stopped working - complete disaster. Took us hours to figure out that one tiny policy change broke the entire stack. The Kubernetes docs mention this behavior switch, but it's buried in paragraph 47 and easy to miss. The CNCF networking patterns explain why this happens, if you enjoy reading academic papers about why your pods can't talk.

Most CNI Plugins Just Ignore Network Policies

Half the CNI plugins don't actually enforce network policies. They'll accept the YAML and do nothing with it.

Flannel doesn't support policies at all. Basic Docker networking? Nope. Whatever your cloud provider installed by default? Maybe, maybe not.

Spent way too much time debugging "broken" policies that were just being ignored:

• Calico: Actually works but debugging sucks
• Cilium: Great debugging tools when everything's working
• Azure CNI: Need the Network Policy Manager addon
• AWS VPC CNI: Needs aws-network-policy-agent which constantly breaks

Test if your CNI actually does anything: create a deny-all policy and see if it blocks connections. If your test pod can still reach external sites, your CNI is ignoring policies. Check the CNI plugin compatibility matrix and Kubernetes conformance tests for your setup.

The Three Ways Network Policies Break

Labels Don't Match

Most policy problems are label mismatches. Your YAML looks fine but selects nothing because:

• Typo: app: frontend vs application: frontend
• Someone changed env: prod to environment: production
• Namespace missing expected labels

Wasted an entire day once because the namespace didn't have the label the policy was looking for. Always check kubectl get pods --show-labels and kubectl get namespaces --show-labels first. The Kubernetes label best practices and selector documentation explain how selectors work.

You Need Both Directions

Network policies are stupid about this - you need BOTH ingress AND egress rules for connections to work. The docs make it sound obvious but it's not.

Frontend needs egress permission to send to backend. Backend needs ingress permission to receive from frontend. Miss either one and connections fail with timeouts that look like any other networking problem.

Debugged this same issue dozens of times. Connection works one way but not the other, or works from A to B but not B to A. Always missing egress rules. The network policy recipes have examples of bidirectional policies that actually work.

DNS Breaks Silently

First network policy breaks DNS but you won't notice immediately. Everything seems fine until someone connects by service name instead of IP.

DNS queries need egress rules to CoreDNS in kube-system. Need both UDP AND TCP port 53 - DNS uses UDP for small responses, switches to TCP for big ones. Miss TCP and DNS randomly fails.

Had a cluster where DNS worked most of the time but randomly broke. Turned out large DNS responses were failing because we only allowed UDP. Intermittent DNS failures are the worst to debug. The DNS troubleshooting guide and CoreDNS documentation cover this in detail.

Cross-Namespace Failures

Multi-namespace apps get ugly fast. Every namespace boundary becomes a firewall.

If your microservices span namespaces, better get all the cross-namespace policies right. Miss one and random services fail in ways that look like any other networking problem. Check out the multi-tenancy patterns and namespace isolation examples for better approaches.

When network policies screw you over, don't waste time reading logs that tell you nothing. Here's what to actually check first, in order of how likely it is to be the problem. I've debugged enough 3am network policy disasters to know this drill by heart.

Step 1: Is Your CNI Plugin Actually Doing Anything?

First thing: check if your CNI gives a damn about network policies. Half the time your "broken" policies are just being ignored:

## Check if you have a real CNI that supports policies
kubectl get pods -n kube-system | grep -E \"(calico|cilium|azure|aws-node)\"

## If you see calico
kubectl get pods -n kube-system -l app=calico-node

## If you see cilium  
kubectl get pods -n kube-system -l k8s-app=cilium

For AWS EKS (which loves to break in creative ways), make sure the network policy agent isn't crashing:

## Check if the AWS network policy agent is actually running
kubectl get pods -n kube-system -l app=aws-node
kubectl logs -n kube-system -l app=aws-node -c aws-network-policy-agent

If you don't see any of these, your CNI doesn't support network policies and you've been wasting your time. Check the CNI specification and plugin compatibility charts to understand what's supported.

Step 2: Find All the Policies That Might Be Screwing You

Network policies are additive, so you need to find ALL of them affecting your pods. There's always more than you think:

## See what policies exist (brace yourself)
kubectl get networkpolicy --all-namespaces

## Look at the specific policies hitting your namespace
kubectl get networkpolicy -n <your-namespace>

## Get the full horror of what's actually configured
kubectl describe networkpolicy <policy-name> -n <namespace>

Check these things or you'll waste hours:

• Pod selectors: Do they actually match your pod labels? Check the selector syntax
• Policy types: If you only have Ingress rules, egress is still blocked (and vice versa)
• Namespace boundaries: Cross-namespace policies need explicit namespace selectors

Multiple policies stack together, so your "simple" deny-all policy plus that "temporary" allow-monitoring policy might be creating unexpected interactions. The policy evaluation order and precedence rules determine what actually gets enforced.

Step 3: Check Your Damn Labels (This is Usually the Problem)

90% of network policy issues are label mismatches. Your policy selectors look right, but they're selecting nothing because labels don't match:

## See what labels your pods actually have (not what you think they have)
kubectl get pods -n <namespace> --show-labels

## Check namespace labels too (everyone forgets these)
kubectl get namespaces --show-labels

## Test if your selector actually matches anything
kubectl get pods -n <namespace> -l app=frontend

## If that returns nothing, your selector is wrong

Common fuckups that will waste your afternoon (and maybe your weekend):

• Typos: app: frontend vs application: frontend
• Case sensitivity: App: Frontend doesn't match app: frontend
• Missing namespace labels: Namespace selectors need labels ON THE NAMESPACE
• Environment drift: Staging uses env: dev but prod uses environment: production

I've debugged 6-hour outages that came down to someone changing tier: web to layer: web in the deployment template. Use label validation webhooks and policy validation tools to catch these before they break production.

Step 4: Actually Test if Things Can Talk to Each Other

Stop guessing and start testing. Here's how to figure out what's actually broken:

## Test direct pod-to-pod (replace with real pod name and IP)
kubectl exec -it frontend-pod -- nc -zv 10.244.1.5 8080

## Test service connectivity (this tests DNS + networking)
kubectl exec -it frontend-pod -- nc -zv backend-service.default.svc.cluster.local 8080

## Test DNS separately (because DNS breaks first)
kubectl exec -it frontend-pod -- nslookup backend-service.default.svc.cluster.local

## Test external connectivity (to rule out total network death)
kubectl exec -it frontend-pod -- nc -zv 8.8.8.8 53

Make a simple connectivity matrix like this:

• Frontend → Backend API: Should work, doesn't work
• Backend → Database: Should work, works
• Frontend → External API: Should work, works
• Backend → DNS: Should work, doesn't work (aha!)

When DNS fails but direct IP works, you know it's DNS egress rules. When direct IP fails, it's the actual network policy blocking traffic. When external connectivity fails, your CNI is probably fucked. The network troubleshooting guide and connectivity debugging tools help narrow down the problem.

Step 5: Check CNI Logs (When You're Desperate)

Each CNI plugin fails in its own special way. Here's how to see what's actually happening:

AWS VPC CNI (Prepare for Pain)

AWS networking logs are scattered across multiple places and the documentation is shit:

## Check if the network policy agent is crashing
kubectl logs -n kube-system -l app=aws-node -c aws-network-policy-agent

## Enable verbose logging (this will spam your logs)
kubectl patch daemonset aws-node -n kube-system --type='merge' -p='{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"aws-network-policy-agent\",\"args\":[\"--enable-policy-event-logs=true\"]}]}}}}'

The logs are usually useless, but sometimes you'll see "policy denied" messages that actually help.

Calico (Cryptic but Functional)

Calico debugging tools are powerful if you can figure out how to use them:

## Check if Calico is actually running
kubectl exec -it -n kube-system <calico-node-pod> -- calicoctl node status

## See what policies Calico thinks it has
kubectl exec -it -n kube-system <calico-node-pod> -- calicoctl get networkpolicy -o wide

Cilium (Great When It Works)

Cilium has the best debugging tools when everything is working:

## See real-time policy decisions
kubectl exec -it -n kube-system <cilium-pod> -- cilium monitor --type=policy-verdict

## Check what endpoints Cilium knows about
kubectl exec -it -n kube-system <cilium-pod> -- cilium endpoint list

Pro tip: If these commands fail, your CNI is probably broken in a fundamental way and you need to restart the CNI pods.

Step 6: The Nuclear Option - Test if Policies Actually Work

If you've gotten this far and nothing makes sense, test whether network policies are doing anything at all:

## Create a test pod that shouldn't be able to reach anything
kubectl run test-external --image=busybox --restart=Never --command -- sleep 3600

## Try to connect to a pod that should be protected
kubectl exec test-external -- nc -zv <protected-pod-ip> <port>

## If this works when it shouldn't, your network policies are being ignored

If the test pod CAN connect when it shouldn't be able to:

• Your CNI doesn't support network policies (go back to step 1)
• You have a policy configuration error that's allowing everything
• The target pod doesn't have any policies applied to it

If the test pod CAN'T connect (good):

• Network policies are working, your issue is elsewhere
• Check label selectors, bidirectional rules, and DNS policies

This is the "hello world" of network policy debugging. If you can't get basic blocking to work, don't bother debugging complex multi-namespace policies.

Once you know what's broken, here's how to actually fix it without making everything worse. These solutions work because I've used them to fix broken production environments at 3am.

Fix #1: Stop Fat-Fingering Your Labels

Label consistency is the #1 cause of network policy pain. Here's how to unfuck your labels:

First, standardize everything NOW:

## Use these exact labels everywhere or you'll hate yourself later
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: frontend              # Keep it simple, no \"application\"
    version: v1.2.1           # For rolling updates
    tier: web                 # web/api/db - nothing fancy
    env: prod                 # prod/staging/dev - short and sweet

Audit your existing mess:

## See all the different app labels in your cluster (prepare to be horrified)
kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.labels.app}{"
"}{end}' | sort | uniq

## Fix them all or you'll be debugging this shit forever

Don't forget namespace labels (everyone does):

apiVersion: v1
kind: Namespace
metadata:
  name: production-frontend
  labels:
    env: prod                 # Match your pod labels exactly
    tier: frontend
    name: production-frontend # Some CNIs need this

Spent most of a day debugging a policy that looked perfect but didn't work. Turns out someone used environment: production in the namespace but env: prod in the pods. Kubernetes label selectors are case-sensitive and exact-match only. Use label standardization tools and policy validation to prevent this.

Fix #2: Unfuck Your DNS (The Thing Everyone Breaks First)

DNS breaks the moment you apply your first network policy. Your apps can reach each other by IP but not by service name. Here's the fix that actually works:

## Copy this exactly - don't get creative
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: your-namespace  # Apply this to EVERY namespace
spec:
  podSelector: {}             # Applies to ALL pods in namespace
  policyTypes:
  - Egress
  egress:
  # Allow DNS to CoreDNS/kube-dns pods (you need both UDP and TCP)
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system   # This label must exist on kube-system namespace
    - podSelector:
        matchLabels:
          k8s-app: kube-dns   # Check what label your DNS pods actually have
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53                # TCP for large DNS responses - don't skip this

Important gotchas that will waste your time:

1. Check your kube-system namespace has name: kube-system label: kubectl get namespace kube-system --show-labels
2. Check your DNS pod labels: kubectl get pods -n kube-system -l k8s-app=kube-dns --show-labels
3. If you're using NodeLocal DNSCache, you need different rules

For NodeLocal DNSCache clusters, add this:

  # Allow access to node-local DNS cache on each node
  - to: []                    # Allows access to any IP (node IPs)
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP  
      port: 53

Had a cluster with intermittent DNS failures for days. Turned out NodeLocal DNSCache wasn't documented anywhere in our setup. Random DNS failures are horrible to debug.

Fix #3: The Bidirectional Policy Dance (Both Directions or It Doesn't Work)

This is where everyone gets confused. You need TWO policies for every connection: one for sending and one for receiving. Miss either one and your app will fail in mysterious ways.

Frontend needs permission to SEND to backend:

## Frontend egress (outbound) policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: frontend-can-call-backend
  namespace: frontend
spec:
  podSelector:
    matchLabels:
      app: web-frontend         # Only apply to frontend pods
  policyTypes:
  - Egress                      # This is OUTBOUND rules
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: backend          # Must match backend namespace label exactly
    - podSelector:
        matchLabels:
          app: api-service       # Must match backend pod labels exactly
    ports:
    - protocol: TCP
      port: 8080                # The actual port your backend listens on

Backend needs permission to RECEIVE from frontend:

## Backend ingress (inbound) policy  
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-allows-frontend
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api-service          # Only apply to backend pods
  policyTypes:
  - Ingress                     # This is INBOUND rules
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: frontend         # Must match frontend namespace label exactly
    - podSelector:
        matchLabels:
          app: web-frontend      # Must match frontend pod labels exactly
    ports:
    - protocol: TCP
      port: 8080                # Same port as above

Pro tip:

Start with just the ingress policy on the backend. If that works, add the egress policy on the frontend. This way you can tell which direction is broken.

Had our entire microservices stack break because of missing egress rules. Connections worked FROM the API gateway but not TO it. Took forever to figure out we only had half the required policies.

Fix #4: CNI-Specific Bullshit That Will Break Your Day

AWS VPC CNI (Expect Pain and Suffering)

AWS VPC CNI network policies are a special kind of broken. The AWS documentation is scattered across 47 different pages and half of it is wrong.

First, check if network policies are even enabled:

## Check if your VPC CNI actually supports network policies (spoiler: maybe)
kubectl describe daemonset -n kube-system aws-node | grep -i network-policy

## Look for PolicyEndpoints CRD (if this doesn't exist, nothing works)
kubectl get crd policyendpoints.networking.k8s.aws

## Check if the network policy agent container is crashing
kubectl logs -n kube-system -l app=aws-node -c aws-network-policy-agent --tail=50

Common AWS VPC CNI failures:

• The network policy agent isn't installed (you need to enable it manually)
• Your cluster doesn't have the right IAM permissions
• Your VPC CNI version is too old (needs 1.14.0+ for network policies)
• PolicyEndpoints aren't being created (check your node group permissions)

Spent way too long debugging a cluster where policies were silently ignored because the addon wasn't installed. AWS doesn't warn you - it just quietly does nothing. Found out after 3 hours that our cluster was running VPC CNI 1.12.0, which predates network policy support by about 6 months.

Calico (Works but the Logs are Cryptic as Hell)

Calico actually enforces network policies, but good luck debugging when they don't work:

Check if Calico is translating your policies correctly:

kubectl exec -n kube-system <calico-node-pod> -- calicoctl get networkpolicy -o wide

## See the actual iptables rules Calico created (prepare for hell)
kubectl exec -n kube-system <calico-node-pod> -- calicoctl get policy --output=yaml

Calico-specific gotchas:

• Calico creates its own policy objects alongside Kubernetes NetworkPolicy objects
• iptables rules are generated per-node and can get out of sync
• Felix logs are verbose but not particularly helpful for debugging policy issues

Cilium (Great Debugging Tools When Everything Works)

Cilium has the best debugging tools, but they only work when Cilium itself is healthy:

Test a specific connection policy decision:

kubectl exec -n kube-system <cilium-pod> -- cilium policy trace --src-k8s-pod=frontend:web-pod --dst-k8s-pod=backend:api-pod --dport=8080

## Monitor policy decisions in real-time (this is actually useful)
kubectl exec -n kube-system <cilium-pod> -- cilium monitor --type=policy-verdict

Cilium-specific gotchas:

• eBPF programs can fail to load on older kernels (check kernel version)
• Cilium policies interact weirdly with kube-proxy in some configurations
• Policy trace only works if you have the exact pod names and namespaces

Fix #5: How to Lock Down Your Cluster Without Breaking Everything

The nuclear option is a default-deny policy that blocks everything, then you allow specific traffic. This works great until you realize you blocked health checks, DNS, API server access, and 47 other things you forgot about.

Here's a default-deny policy that doesn't completely destroy your cluster:

## Copy this but test it in staging first (seriously)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-but-allow-basics
  namespace: production                    # Apply to each namespace separately
spec:
  podSelector: {}                          # Affects ALL pods in namespace
  policyTypes:
  - Ingress                                # Block all inbound
  - Egress                                 # Block all outbound
  egress:
  # Allow DNS (or everything breaks immediately)
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
  # Allow Kubernetes API access (for health checks and service discovery)
  - to: []                                 # Any destination
    ports:
    - protocol: TCP
      port: 443                            # Kubernetes API server port
  # Allow common health check ports (adjust for your apps)
  - to: []
    ports:
    - protocol: TCP
      port: 8080                           # Common health check port
    - protocol: TCP
      port: 9090                           # Prometheus metrics port

Things this policy will still break:

• Anything that uses non-standard health check ports
• Applications that need to talk to external services
• Monitoring that scrapes metrics on non-standard ports
• Any inter-pod communication you haven't explicitly allowed

Test this everywhere first. Seen this policy break entire environments because monitoring used port 9100 instead of 9090, or some random app needed a weird port nobody documented.

Advanced Troubleshooting for Complex Policy Interactions

Policy Overlap Resolution

When multiple policies apply to the same pods, understanding their combined effect requires systematic analysis:

## List all policies affecting a specific pod
kubectl get networkpolicy --all-namespaces -o json | jq -r '.items[] | select(.spec.podSelector.matchLabels | keys | length > 0) | .metadata.namespace + "/" + .metadata.name'

## Create policy effectiveness matrix
for policy in $(kubectl get networkpolicy -n <namespace> -o name); do
  echo "=== $policy ==="
  kubectl describe $policy -n <namespace>
done

When Your Cluster is Slow as Hell

Too many network policies can make your cluster perform like shit. Each policy creates iptables rules or eBPF programs, and too many rules slow down packet processing.

Performance fixes:

1. Consolidate policies: If you have 5 policies with the same selectors, combine them
2. Use namespace selectors: One policy for a whole namespace beats 20 pod-specific policies
3. Monitor your CNI: Check if Calico Felix or Cilium agent is eating CPU

Test Your Policies Before They Break Production

Write simple tests so you know if your policies actually work:

#!/bin/bash
## Test if your policy actually blocks what it should block
test_connection() {
  local source_pod=$1
  local target_host=$2
  local target_port=$3
  local should_work=$4
  
  echo "Testing: $source_pod -> $target_host:$target_port"
  if kubectl exec $source_pod -- nc -zv $target_host $target_port 2>/dev/null; then
    result="WORKS"
  else
    result="BLOCKED"
  fi
  
  if [ "$result" = "$should_work" ]; then
    echo "✓ PASS: $result (expected $should_work)"
  else
    echo "✗ FAIL: $result (expected $should_work)"
    return 1
  fi
}

## Test that frontend can reach backend
test_connection "frontend-pod" "backend-service" "8080" "WORKS"

## Test that frontend cannot reach database directly
test_connection "frontend-pod" "database-service" "5432" "BLOCKED"

Run these tests every time you change policies. Trust me, you'll catch policy mistakes before they hit production.