Docker Exit Code 137: OOM Kill Prevention and Debugging

Critical Context

What Exit Code 137 Means:

• Signal: 128 + 9 (SIGKILL - uncatchable termination)
• Cause: Linux kernel OOM killer terminated container for exceeding memory limits
• Timing: Often occurs during traffic spikes or at 3AM when monitoring is minimal
• Impact: No graceful shutdown, no cleanup, immediate service disruption

Common Production Scenario:
Container runs stable for weeks with 512MB limit → Traffic spike requires 600MB → Instant death with no warning

Configuration That Actually Works

Memory Limit Sizing Formula

Memory limit = (Peak RSS × 1.5) + Runtime overhead

Runtime Overhead Requirements:

• JVM applications: +200-400MB for non-heap memory
• Node.js applications: +50-100MB for V8 overhead
• Go applications: +20-50MB for garbage collection
• Python applications: +30-100MB for interpreter overhead

JVM Container Configuration

Critical Problem: JVM allocates heap based on host memory (32GB), not container limits (512MB)

Required Flags:

# Modern JVMs (Java 11+)
-XX:+UseContainerSupport

# Legacy JVMs
-Xmx400m  # Leave 112MB for non-heap in 512MB container

Failure Mode: JVM tries to allocate 8GB heap in 512MB container → immediate OOM kill

Kubernetes Memory Settings

2025 Best Practice: Set requests = limits to prevent overcommitment

Why This Matters:

• Scheduler uses requests for node placement
• OOM killer respects limits
• Gap between them causes unpredictable failures when nodes are overcommitted

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "512Mi"  # Same value prevents overcommit

Overcommitment Failure: Multiple pods scheduled based on low requests, all hit high limits simultaneously → node memory exhaustion → random pod kills

Diagnostic Commands

Confirm OOM Kill

# Check OOM kill flag (can be misleading for child process kills)
docker inspect --format '{{.State.OOMKilled}}' container_name

# Verify exit code
docker inspect --format '{{.State.ExitCode}}' container_name

# Check system logs for actual OOM events
dmesg | grep -i "killed process"

Memory Monitoring

# Real-time monitoring (includes cache/buffers - can be misleading)
docker stats container_name

# Container-internal memory view
docker exec container_name cat /proc/meminfo

Critical Warning: docker stats shows cache memory that kernel can reclaim. OOM killer looks at RSS (resident set size). Container can show 200MB usage but die at 512MB limit due to committed memory.

Failure Scenarios and Solutions

Memory Leak vs Memory Spike Detection

Memory Leak Pattern:

• Gradual memory growth over time
• No periodic drops from garbage collection
• Consistent upward trend in monitoring

Memory Spike Pattern:

• Stable baseline usage
• Sudden jumps during traffic/processing
• Returns to baseline after load decreases

Diagnostic Difference: Leaks require application profiling, spikes require better resource planning or rate limiting.

Child Process OOM Confusion

Scenario: Container shows OOMKilled: false but exit code 137
Cause: Child process got OOM killed, main process (PID 1) remained alive but exited
Detection: Check kernel logs for actual OOM events, not just Docker flags

False OOM Signals

Windows Containers: OOM behavior differs significantly from Linux
Child Process Kills: Docker only flags OOMKilled if PID 1 dies directly
Memory Accounting: Different between Docker stats and kernel OOM killer view

Production Prevention Strategies

Application-Level Defense

Backpressure Implementation:

• Reject non-essential operations at 80% memory usage
• Implement circuit breakers for memory-intensive operations
• Stream data processing instead of loading entire datasets

Connection Pool Limits:

# Memory-aware pool sizing
session = requests.Session()
session.mount('http://', HTTPAdapter(pool_maxsize=50))

Monitoring Setup Requirements

Essential Stack:

• cAdvisor for accurate container metrics collection
• Prometheus for storage and alerting
• Alert at 75% memory usage, panic at 90%
• Monitor memory growth rate, not just absolute usage

Critical Metric: RSS memory tracking, not Docker stats cache-inclusive numbers

Container Sizing Methodology

1. Baseline Measurement: Run production workload without limits for 48-72 hours
2. Peak Calculation: Monitor actual RSS usage under realistic load
3. Safety Margin: Apply 1.5x multiplier plus runtime overhead
4. Load Testing: Verify limits under stress conditions that mimic production spikes

Exit Code Reference

Code	Meaning	Required Action
137	SIGKILL (often OOM)	Check OOMKilled flag, increase memory or fix leak
125	Docker daemon error	Verify Dockerfile syntax, image availability
126	Command not executable	Fix permissions or command path
127	Command not found	Check binary exists, PATH configuration
1	Application error	Review application logs for specific error
0	Clean exit	Normal termination (investigate if unexpected)

Resource Requirements

Time Investment

• Initial Sizing: 2-3 days of monitoring plus load testing
• Production Debugging: 30 minutes to 4 hours depending on complexity
• Monitoring Setup: 1-2 days for complete observability stack

Expertise Requirements

• Basic: Understanding of container memory limits and Docker commands
• Advanced: Knowledge of kernel memory management, cgroups, and runtime-specific behavior
• Expert: Application profiling, custom metrics, and distributed system debugging

Breaking Points

• 1000+ concurrent containers: Standard monitoring tools may become inadequate
• Multi-GB containers: Require careful node sizing and network considerations
• High-frequency allocations: May need custom memory management strategies

Critical Warnings

What Documentation Doesn't Tell You

• Docker stats memory reporting includes reclaimable cache
• OOMKilled flag only set when PID 1 dies directly
• JVM heap size calculation ignores container limits by default
• Kubernetes scheduler and OOM killer use different memory values
• Child process OOM kills don't trigger container restart policies

Hidden Costs

• Memory Overcommitment: Appears to save resources but causes unpredictable failures
• Insufficient Monitoring: Delayed detection leads to extended outages
• Undersized Containers: Create cascade failures during traffic spikes
• Legacy Runtime Defaults: Most runtimes ignore container memory limits without explicit configuration

Migration Pain Points

• Kubernetes 1.20+: Changes in memory accounting affect existing deployments
• Docker Desktop vs Production: Different memory management behavior
• Cloud Platform Differences: AWS ECS, Azure Container Apps, GCP Cloud Run have platform-specific OOM handling

This knowledge enables automated detection of memory pressure, proper container sizing, and prevention of production OOM kills through systematic monitoring and application-level defensive programming.