Docker Container Performance Optimization - AI Technical Reference

Critical Failure Scenarios

OOMKilled Containers (Exit Code 137)

• Cause: Container exceeds memory limits, kernel kills process
• Detection: kubectl get events --all-namespaces | grep "OOMKilling"
• Impact: Production service downtime, cascading failures
• Root Cause: Docker Desktop doesn't enforce memory limits properly - production Kubernetes does
• Solution: Test locally with docker run --memory=512m your-app

Image Size and Startup Performance

• Problem Scale: Images 2GB+ common, startup times 30-60 seconds
• Real Impact: Network transfer delays, security scan overhead, deployment bottlenecks
• Cost Impact: AWS bills can increase from $1,200 to $8,000/month due to oversized instances
• Solution Impact: Multi-stage builds reduce images from 1.8GB to ~200MB, startup from 60s to 8-10s

Container Networking Latency

• Overhead: Service mesh adds 100-200ms per request
• Common Failure: Istio health checks every 500ms, timing out at 200ms for 300ms database queries
• Fix Impact: Tuning health checks from 100ms/200ms timeout to 5s intervals reduced P95 latency by 80%

Resource Requirements and Limits

Memory Allocation Strategy

Application Type	Normal Usage	Spike Usage	Recommended Limit	Reasoning
Node.js API	150MB	300MB	400MB	2x normal for traffic spikes
Java Spring Boot	800MB	1.2GB	1.5GB	JVM heap + GC overhead
Python Flask	80MB	120MB	200MB	Conservative for memory leaks

CPU Resource Guidelines

• Requests: Set to average usage under load
• Limits: 2-3x requests for burst capacity
• Critical Warning: CPU throttling is silent - pods slow down without obvious errors
• Java Special Case: Use -XX:+UseContainerSupport -XX:MaxRAMPercentage=75 to respect container limits

Image Optimization Strategies

Multi-Stage Build Template

# Build stage - contains build tools and dependencies
FROM node:18-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production --no-audit --no-fund
RUN npm prune --production

# Runtime stage - minimal production image
FROM node:18-alpine AS runtime
RUN addgroup -g 1001 -S nodejs && adduser -S nodejs -u 1001
WORKDIR /app
COPY --from=build --chown=nodejs:nodejs /app/node_modules ./node_modules
COPY --chown=nodejs:nodejs . .
USER nodejs
EXPOSE 3000
CMD ["node", "server.js"]

Results: 180MB final images, 8-10s startup times on AWS ECS

Base Image Selection

Image Type	Size	Compatibility	Use Case	Failure Rate
Alpine Linux	5MB	DNS/glibc issues common	High maintenance tolerance	High
Debian Slim	50MB	Full compatibility	Production recommended	Low
Distroless	Variable	No debug capability	Security-critical apps	Medium
Ubuntu	200MB+	Full compatibility	Legacy apps only	Low

Alpine Linux Warning: Uses musl libc instead of glibc - causes DNS race conditions, PostgreSQL connection failures, Python package breakage

Network Performance Optimization

Service Mesh Impact

• Latency Overhead: 100-200ms per request minimum
• Health Check Failures: Default timeouts too aggressive for real applications
• Alternative: Use Kubernetes native services for internal communication
• When to Use: Only if traffic splitting or mutual TLS required

DNS Performance Issues

• Container DNS Overhead: 50-100ms per lookup
• Solution: Cache DNS for 30+ seconds
• Node.js Fix: dns.setDefaultResultOrder('ipv4first')
• Impact: Response times dropped from 500ms to 50ms with DNS caching

Storage and Logging Anti-Patterns

Critical Storage Rules

• Never: Log to files inside containers - fills filesystem, causes write failures
• Always: Log to stdout/stderr only
• Database Storage: Use persistent volumes, not bind mounts (15-20% faster)
• Temporary Files: Use emptyDir volumes

Real Disaster Example

• PostgreSQL container filled 20GB volume with logs in 3 days
• Database went read-only
• Application crashed
• Required Saturday restoration from backup
• Fix: Ship logs to CloudWatch, rotate daily

Monitoring and Alerting Configuration

Essential Metrics

1. Pod Restart Rate: rate(kube_pod_container_status_restarts_total[10m]) > 0
2. Memory Usage: container_memory_usage_bytes / container_spec_memory_limit_bytes * 100 > 85
3. Request Latency P95: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1

Alert Thresholds

• Pod Restarts: >3 restarts in 10 minutes (indicates OOMKills, crashes)
• Response Time: P95 >1 second (performance degradation)
• Error Rate: >1% (application issues)
• Memory Usage: >85% (approaching OOMKill)

Platform-Specific Considerations

AWS Graviton (ARM64)

• Cost Savings: 40% cheaper than x86
• Compatibility Risk: Many Docker images lack ARM64 support
• Requirement: Multi-arch image builds
• Performance: Significant improvements for most workloads

BuildKit Cache Issues

• Problem: Cache corruption occurs weekly in CI environments
• Symptom: 5-minute builds become 20-minute full rebuilds
• Workaround: docker builder prune -f in daily CI pipeline
• Alternative: External cache storage (S3), though Docker caching remains unreliable

Autoscaling Configuration

HPA Settings That Prevent Flapping

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  minReplicas: 2
  maxReplicas: 20
  targetCPUUtilization: 80
  scaleDownPolicy:
    stabilizationWindowSeconds: 300  # Prevents constant scaling

Rationale: 80% CPU threshold scales before user impact, 5-minute scale-down prevents oscillation

Security and Compliance

Container Security Requirements

• User Permissions: Never run as root - use USER nodejs directive
• Base Images: Keep Docker updated for container escape patches
• Image Scanning: Automated scanning catches known vulnerabilities
• Network Policies: Implement without service mesh overhead

Cost Optimization Strategies

Immediate Cost Reduction

1. Multi-stage builds: 5x smaller images = lower storage/transfer costs
2. Resource right-sizing: Monitor for 1 week, set limits to 2x observed usage
3. Spot instances: 70% cost reduction (with availability risk)
4. ARM64 instances: 40% cost reduction (with compatibility requirements)

Cost Monitoring

• Kubernetes resource allocation: kubectl describe node | grep -A 5 "Allocated resources"
• Pod resource usage: kubectl top pods --all-namespaces --sort-by=memory
• Budget alerts: Set up before bills reach CFO attention

Implementation Priority

Phase 1 - Immediate Fixes (2-4 hours)

1. Identify OOMKilled pods: kubectl get events | grep OOMKilling
2. Monitor resource usage: kubectl top pods
3. Fix largest images with multi-stage builds

Phase 2 - Resource Optimization (1 week)

1. Set proper resource limits based on monitoring
2. Implement health checks with realistic timeouts
3. Configure autoscaling with anti-flapping measures

Phase 3 - Advanced Optimization (2-3 weeks)

1. Implement comprehensive monitoring with Prometheus/Grafana
2. Optimize networking (remove unnecessary service mesh)
3. Platform-specific optimizations (ARM64, spot instances)

Troubleshooting Commands

Diagnostic Commands

# Memory usage by pod
kubectl top pods --all-namespaces --sort-by=memory

# Find restarting pods (OOMKills)
kubectl get pods --all-namespaces --field-selector=status.phase=Running | grep -v " 0 "

# Check node resource allocation
kubectl describe node | grep -A 5 "Allocated resources"

# Recent OOMKill events
kubectl get events --all-namespaces | grep "OOMKilling"

# CPU throttling detection
kubectl top pods --containers --all-namespaces

Performance Analysis

• Image layer analysis: Use dive tool to identify bloated layers
• Resource monitoring: Deploy Prometheus + Grafana (200-500MB RAM overhead per node)
• Network latency: Check service mesh configuration and health check timings

Critical Warnings

What Official Documentation Doesn't Mention

• Docker Desktop: Doesn't enforce memory limits - production will
• Alpine Linux: DNS race conditions affect production reliability
• BuildKit: Cache corruption requires weekly manual intervention
• Java Containers: JVM ignores container limits without specific flags
• Service Mesh: Adds significant latency overhead for marginal security benefits

Decision Criteria

• Use Alpine: Only if debugging DNS failures at 3 AM sounds manageable
• Use Service Mesh: Only if traffic splitting or mutual TLS absolutely required
• Use Spot Instances: If application can handle random termination
• Use ARM64: If all Docker images support multi-arch builds

Success Metrics

• Image Size: Target <200MB for typical applications
• Startup Time: Target <15 seconds in production
• Memory Efficiency: 2x headroom over observed usage
• Cost Reduction: 40-60% savings possible with proper optimization
• Reliability: <3 pod restarts per 10 minutes indicates stable configuration