Docker Container Health Check Debugging - AI-Optimized Guide

Configuration

Production-Ready Health Check Settings

HEALTHCHECK --start-period=60s --interval=30s --timeout=10s --retries=3 \
  CMD curl -f localhost:8080/health

Critical Timing Parameters:

• start-period=60s: Most apps need 30-60 seconds minimum startup time
• timeout=10s: Don't wait forever - health checks taking >10s indicate problems
• retries=3: Prevents single random timeout from causing alerts
• interval=30s: Sweet spot between resource usage and detection speed

Database-Specific Health Checks

• PostgreSQL: pg_isready -h localhost
• MySQL: mysqladmin ping -h localhost
• Redis: redis-cli ping

Warning: Never use expensive database queries in health checks (e.g., SELECT COUNT(*) FROM huge_table)

Common Failure Modes and Solutions

90% Failure Rate Causes

1. Wrong port configuration: Health check hits port 3000, app runs on 8080
2. Missing curl in container: Exit code 125 - command not found
3. localhost vs 0.0.0.0: App listening on 127.0.0.1, health check targets localhost
4. Insufficient startup time: Default 0-second start-period kills slow-starting apps

Network Configuration Issues

• Container listening on localhost: Change app to listen on 0.0.0.0:PORT
• Port verification: docker exec container netstat -tlnp
• DNS problems: Use IP addresses instead of hostnames
• IPv6 conflicts: Force IPv4 with curl -4

Resource Constraints

• Memory limits: Health checks timeout during OOM conditions
• Detection: docker stats container - watch for memory approaching limits
• CPU starvation: Random timeouts under load
• Garbage collection pauses: Can cause intermittent health check failures

Critical Debugging Commands

Health Check Status Investigation

# Get complete health check state
docker inspect --format "{{json .State.Health }}" container | jq

# View health check logs
docker inspect --format "{{json .State.Health }}" container | jq '.Log[].Output'

# Manual health check execution
docker exec -it container curl -f localhost:8080/health
echo $?

Exit Code Meanings

• 0: Success
• 1: General failure (app broken)
• 125: Docker couldn't run command (missing dependencies)
• Timeout: No exit code - Docker kills hanging process

Environment Replication

# Replicate exact Docker health check environment
docker exec container sh -c "curl -f localhost:8080/health"

Resource Requirements

Time Investment

• Initial debugging: 2 hours typical for first-time issues
• Recurring problems: 30 minutes once patterns identified
• Prevention setup: 1 hour to configure proper health checks

Expertise Requirements

• Basic Docker knowledge: Essential
• Application architecture understanding: Critical for meaningful health checks
• Network troubleshooting: Required for complex failures

Implementation Reality

Docker Compose Dependency Management

services:
  web:
    depends_on:
      database:
        condition: service_healthy  # Critical: prevents startup race conditions
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      start_period: 30s

  database:
    image: postgres:13
    healthcheck:
      test: ["CMD", "pg_isready", "-h", "localhost"]
      interval: 10s
      timeout: 5s
      retries: 5

Health Check Endpoint Requirements

Must Test:

• Database connectivity (if app can't function without it)
• Critical cache availability
• Essential external dependencies

Must NOT Test:

• Non-critical external APIs (causes false failures)
• Expensive operations (slows down app)
• File system operations (unreliable timing)

Breaking Points and Failure Modes

False Positive Scenarios

• Tuesday 2am mystery: Automated maintenance consuming resources
• Load-dependent failures: Health checks work when idle, fail under traffic
• Docker version upgrades: Behavior changes between versions
• Environment differences: Works locally, fails in CI/CD

Critical Warnings

• Container restart loops: Broken health checks can prevent proper startup
• Resource monitoring gaps: Health check failures often mask underlying resource issues
• Default setting traps: Docker's 30-second defaults fail for most real applications
• Orchestration confusion: Health check success ≠ application readiness

Monitoring Requirements

Essential Metrics

• Health check success rate over time
• Health check response times
• Resource usage correlation with failures
• Failure pattern analysis (time, load conditions)

Alert Configuration

• Don't alert on: Single health check failure
• Alert on: 3+ consecutive failures (matches Docker's retry logic)
• Escalate on: Persistent failures >10 minutes

Decision Criteria

When Health Checks Are Worth It

• Multi-service applications: Dependencies must start in order
• Production environments: Automated recovery essential
• Load-balanced deployments: Traffic routing decisions

When to Skip Health Checks

• Single-container applications: Process monitoring sufficient
• Development environments: Manual intervention acceptable
• Stateless services: Container restart has no side effects

Production Gotchas

Container State Confusion

• Container status "running" + health status "unhealthy" = app process alive but non-functional
• Health check failure ≠ container restart (orchestration system dependent)
• Docker doesn't restart unhealthy containers automatically

Testing Requirements

# Local validation before deployment
docker build -t myapp .
docker run -d --name test-container myapp
sleep 60
docker inspect --format "{{json .State.Health }}" test-container | jq

Monitoring Commands

# Real-time health status changes
docker events --filter event=health_status

# Resource correlation analysis
docker stats container

Migration Pain Points

Common Upgrade Issues

• Health check timing behavior changes between Docker versions
• Container orchestration system updates modify health check handling
• Base image updates may remove health check dependencies (curl, netstat)

Validation Checklist

• Test health checks in target deployment environment
• Verify health check dependencies exist in container
• Confirm timing settings match application startup requirements
• Validate network configuration matches health check expectations