Docker Swarm Service Discovery & Routing Mesh Failure Guide

Critical System Overview

Docker Swarm networking consists of 5 interdependent layers that must all function correctly:

• Embedded DNS server (127.0.0.11)
• VXLAN overlay networks (UDP port 4789)
• IPVS load balancing (Linux kernel)
• Certificate-based node authentication
• Routing mesh for published ports

Failure Impact: When any layer fails, entire distributed applications become unreachable despite individual containers showing healthy status.

Common Failure Patterns

DNS Resolution Failures

Symptoms:

• "Service not found" errors when containers are running
• Empty results from tasks.<service-name> queries
• Intermittent connection failures (30-80% failure rate)

Root Causes:

• Embedded DNS server returning stale container IPs from dead containers
• DNS performance degradation under load (>200 concurrent requests)
• Cross-node communication failures preventing DNS synchronization

Resolution Time: 5-30 minutes for DNS cache refresh, 2-6 hours for overlay network reconstruction

MTU Fragmentation Issues

Critical Threshold: VXLAN adds 50 bytes overhead - networks with 1500 MTU will drop packets >1450 bytes

Symptoms:

• Ping works (64-byte packets) but HTTP requests fail
• File uploads randomly fail
• Database queries timeout intermittently

Production Impact: $2000/minute revenue loss during Black Friday traffic when large API responses fail

Fix: Set MTU to 1450 permanently in /etc/docker/daemon.json

VXLAN Tunnel Failures

Port Requirements:

• TCP 2377 (manager communication)
• TCP/UDP 7946 (node communication)
• UDP 4789 (overlay network data)

Conflict Sources:

• VMware NSX using same port 4789
• Corporate firewalls blocking UDP traffic
• Cloud security groups misconfigured

Workaround: Use --data-path-port=7789 when initializing swarm

Diagnostic Decision Tree

Step 1: Service vs Infrastructure Failure

# Deploy test service across all nodes
docker service create --name connectivity-test --mode global --publish 8999:80 nginx:alpine

• Test passes: Application-specific issue
• Test fails: Infrastructure networking failure

Step 2: DNS Layer Testing

# From inside container
nslookup <service-name>        # VIP resolution
nslookup tasks.<service-name>  # Individual container IPs
nslookup google.com           # External DNS validation

Failure Patterns:

• VIP works, tasks fail = Task discovery broken
• Both fail = DNS completely broken
• External fails = Container DNS configuration corrupted

Step 3: Network Layer Validation

# Test packet size limits
ping -s 1400 <remote-node-ip>  # Should work
ping -s 1472 <remote-node-ip>  # Will fail if MTU misconfigured

Step 4: Load Balancer State Check

# Check for stale backend entries
sudo ipvsadm -L -n --stats

Red flags: Backends pointing to non-existent containers, uneven connection distribution

Resource Requirements

Time Investment by Issue Type

• DNS cache refresh: 5-15 minutes
• MTU reconfiguration: 15-30 minutes + service restart
• Overlay network recreation: 1-3 hours + planned downtime
• Certificate rotation: 30-60 minutes
• Complete cluster rebuild: 4-8 hours + full service migration

Expertise Requirements

• Basic troubleshooting: Understanding of DNS, TCP/IP fundamentals
• Advanced debugging: Linux networking, iptables, VXLAN protocol knowledge
• Cluster recovery: Docker Swarm architecture, certificate management

Infrastructure Dependencies

• Monitoring tools: netshoot container, Weave Scope for network visualization
• Root access: Required for IPVS commands, firewall configuration
• Network access: All diagnostic commands require SSH/direct access to nodes

Critical Configuration Settings

Production-Ready daemon.json

{
  "mtu": 1450,
  "live-restore": true,
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Required Firewall Rules

# Manager nodes
ufw allow from <cluster-subnet> to any port 2377

# All nodes
ufw allow from <cluster-subnet> to any port 7946
ufw allow from <cluster-subnet> to any port 4789/udp

System Resource Limits

# /etc/systemd/system/docker.service.d/override.conf
[Service]
LimitNOFILE=1048576
LimitNPROC=1048576
TasksMax=infinity

Failure Recovery Procedures

Emergency DNS Reset (5-10 minutes)

# Force DNS cache refresh
docker service update --force <service-name>

# Aggressive reset if needed
sudo systemctl restart docker

MTU Fix (15-30 minutes)

# Immediate fix
sudo ip link set dev docker_gwbridge mtu 1450

# Permanent configuration
echo '{"mtu": 1450}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker

Complete Network Reconstruction (2-4 hours)

# 1. Export service configurations
docker service ls --format "{{.Name}}" | xargs -I {} docker service inspect {} > backup.json

# 2. Scale services to 0
docker service ls --format "{{.Name}}" | xargs -I {} docker service scale {}=0

# 3. Remove overlay networks
docker network ls --filter driver=overlay --format "{{.Name}}" | grep -v ingress | xargs docker network rm

# 4. Restart Docker cluster-wide
sudo systemctl restart docker

# 5. Recreate and scale back up

Monitoring and Prevention

Early Warning Indicators

• DNS query response time >5 seconds (normal: <100ms)
• IPVS backend connection count showing 0 for healthy containers
• Docker daemon memory usage >80% of available RAM
• Certificate expiration within 30 days

Automated Health Checks

# DNS resolution monitoring
docker service create --name dns-monitor --mode global \
  alpine sh -c 'while true; do time nslookup tasks.web; sleep 60; done'

# Cross-node connectivity testing
docker service create --name connectivity-monitor --mode global \
  alpine sh -c 'while true; do curl -f "${SERVICE_NAME}:${PORT}/health"; sleep 30; done'

Known Breaking Points

Scale Thresholds

• DNS performance: Degrades significantly above 200 concurrent services
• IPVS state: Becomes unreliable with >1000 backend changes per hour
• Certificate rotation: Requires cluster downtime above 50 nodes

Environmental Limitations

• Geographic distribution: Cross-region latency makes VXLAN unstable
• Enterprise networks: Corporate firewalls frequently block UDP 4789
• Cloud platforms: Security group defaults often break node communication
• Virtualization: VMware NSX conflicts with Docker VXLAN implementation

Version-Specific Issues

• Docker 19.03+: DNS server memory leaks under high load
• Docker 20.10.8: Connection pool failures with tasks. DNS queries
• Linux kernel 5.4+: IPVS connection tracking changes affect load balancing

Decision Criteria

When to Use Docker Swarm vs Alternatives

Use Docker Swarm when:

• <50 nodes in single datacenter
• Simple service discovery requirements
• Minimal networking customization needed

Consider alternatives when:

• Multi-region deployment required
• Complex networking policies needed
• High-availability SLA >99.9%
• Team lacks deep Docker networking expertise

Troubleshooting Cost-Benefit Analysis

Worth immediate fix:

• MTU configuration (high success rate, low risk)
• DNS cache refresh (quick, non-disruptive)
• Certificate rotation (addresses authentication failures)

Consider alternatives before attempting:

• Complete overlay network rebuild (high downtime risk)
• Multi-node certificate reset (requires coordination)
• Cluster-wide Docker restart (service interruption guaranteed)

Common Misconceptions

1. "Restarting Docker fixes everything" - Only resolves DNS cache issues, not underlying network problems
2. "Ping success means networking is fine" - ICMP uses small packets, doesn't test MTU or application protocols
3. "Container health checks validate service discovery" - Health checks bypass Docker networking layer
4. "Default settings work in production" - Default MTU 1500 causes VXLAN fragmentation in most environments
5. "Load balancing is automatic" - IPVS state corruption requires manual intervention

Emergency Contacts and Escalation

When to Escalate

• Multiple overlay network reconstruction attempts failed
• Certificate issues affecting >50% of cluster
• Network partitions lasting >30 minutes
• Data corruption in IPVS state requiring kernel-level intervention

Required Information for Support

• Docker version and kernel version
• Network topology diagram
• Complete output of docker system info
• Service configurations and placement constraints
• Firewall rules and security group configurations
• Recent infrastructure changes or deployments