Stop Your Docker Containers From Eating Production Alive

Three months ago, our Kubernetes cluster started randomly killing pods with exit code 137 (OOMKilled). Staging was fine, but prod was falling apart completely. Turns out, nobody set memory limits properly, and some Node.js app was consuming memory like crazy - I think it hit around 8GB or maybe more before Kubernetes murdered it. Someone was storing entire HTTP request bodies in memory like a fucking amateur.

The Real Problems Nobody Talks About

Docker Desktop on your MacBook is not production. I don't care how many times you've run docker-compose up - production will find new and exciting ways to break your shit.

Problem #1: Your Images Are Obese
That Dockerfile you copied from Stack Overflow? It's pulling Ubuntu, installing like 400MB of build tools, and leaving everything behind. My team was deploying massive Node.js images - over 2GB - until we figured out multi-stage builds. Now they're around 180MB, maybe less. Startup time dropped from almost a minute down to maybe 8 or 10 seconds on AWS ECS.

Problem #2: Memory Limits Will Kill You
Set your memory limits too low? Kubernetes kills your pods. Set them too high? Your AWS bill explodes. We learned this the hard way when our bill went from roughly twelve hundred bucks to over eight grand in one month because nobody set resource constraints. The logs just said "Pod killed" - no helpful details, as fucking usual.

Worth noting: Kubernetes resource monitoring has gotten somewhat better recently, but you still need to dig through verbose kubectl describe node output to figure out why pods are getting throttled. The logs just say "resource pressure" which isn't exactly helpful for debugging.

Problem #3: Networking Is Black Magic
Container networking adds overhead you don't expect. We saw latency spikes around 200ms between services that should have been under 10ms. Turns out, our service mesh (Istio) was doing health checks every 500ms and failing like half of them. The solution? Tune the health check timeouts and use host networking for latency-critical services.

The Shit That Actually Matters

Forget the "comprehensive optimization strategies" - here's what fixed our production nightmare:

Alpine Linux Breaks Everything
Alpine uses musl instead of glibc. A bunch of your dependencies will break in mysterious ways. We spent probably two days debugging why our Python app couldn't connect to PostgreSQL before finding this GitHub issue. The Alpine security vulnerabilities are also an ongoing problem. Stick with Debian slim unless you enjoy debugging DNS resolution failures at 3 AM.

BuildKit Cache Is Trash
Docker's BuildKit caching randomly corrupts and forces full rebuilds. We added docker builder prune -f to our CI pipeline because it happens like clockwork every week or so. Your 5-minute builds turn into 20-minute builds, and Docker's error messages are about as helpful as a screen door on a submarine. The BuildKit GitHub issues are full of people with the same exact problem.

Security reminder: Keep Docker updated. Container escapes happen, and you don't want to be the person explaining to the security team how an attacker broke out of a container because you're running last year's Docker version.

Java Containers Are Special Snowflakes
Java in containers is a pain in the ass. The JVM doesn't understand container memory limits by default. Use `-XX:+UseContainerSupport` flag and set `-Xmx` to 75% of your container limit, or the JVM will try to use all available system memory and get killed by the OOM killer.

What Actually Works

After wasting months on "best practices" that don't work, here's what saved our production:

1. Use distroless images - Google's distroless images have no shell, no package manager, nothing. Your attack surface is tiny, and startup is fast. Check out the distroless security benefits if you care about compliance.

2. Set proper resource limits - If you don't know what your app needs, start with 200m CPU and 512Mi memory. Monitor for a week, then adjust. Better to start conservative than kill your budget. The Kubernetes resource management docs have the details.

3. Enable proper health checks - Kubernetes needs to know when your app is ready. Use /health endpoints that actually check your database connections, not just return HTTP 200. Proper liveness and readiness probes prevent cascading failures.

The bottom line? Your development environment lies to you. Production will find every edge case, race condition, and performance bottleneck you didn't know existed. Container resource monitoring is essential, proper health checks prevent cascading failures, and troubleshooting guides become your best friends at 3 AM. Plan for failure, set proper limits, and always have a rollback strategy ready.

You get why production hates your containers. Here's what's probably broken right now:

Multi-Stage Builds: From Massively Bloated to Actually Reasonable

The Dockerfile we inherited was a crime against humanity. It was based on Ubuntu 20.04, installed the entire build toolchain, downloaded like 500MB or more of dependencies, and left it all in the final image. Our Node.js app was over 2GB - bigger than most fucking operating systems.

Here's what fixed it:

## Build stage - all the shit you need to compile
FROM node:18-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production --no-audit --no-fund
RUN npm prune --production

## Runtime stage - only what you need to run
FROM node:18-alpine AS runtime
RUN addgroup -g 1001 -S nodejs && adduser -S nodejs -u 1001
WORKDIR /app
COPY --from=build --chown=nodejs:nodejs /app/node_modules ./node_modules
COPY --chown=nodejs:nodejs . .
USER nodejs
EXPOSE 3000
CMD ["node", "server.js"]

Result: Final image came out around 180MB, startup time dropped from almost a minute down to maybe 8-10 seconds on AWS ECS. Don't use Ubuntu unless you actually need systemd or specific packages. Alpine works for most use cases, when it doesn't break everything.

Warning: That USER node line? It'll break your app if it tries to write to /app or bind to port 80. Test locally with the user switch first.

Resource Limits: Stop Guessing, Start Measuring

Everyone copies resource limits from Stack Overflow without understanding what their app actually needs. I spent a week profiling our apps with Kubernetes resource monitoring before setting limits.

Real numbers from our production setup:

• Node.js API: around 150MB normal usage, spikes to maybe 300MB → we set limits to 400MB
• Java Spring Boot: roughly 800MB baseline, spikes up to 1.2GB or more → set limit to 1.5GB
• Python Flask: typically 80MB, peaks around 120MB → limit set to 200MB

Check Kubernetes memory assignment docs and resource resizing for the details on how to actually set these.

CPU is weird: Set requests to what you need under load, limits to 2-3x that. If you set CPU limits too low, your app gets throttled and responds slower than a hungover intern.

Use kubectl top pods to see actual usage. If your pods get OOMKilled, your limits are too low. If your AWS bill is astronomical, your limits are too high.

Don't just copy-paste this nuclear option shit. Figure out what's actually broken first:

Networking: The Invisible Performance Killer

Container networking is like TCP congestion control - it works until it doesn't, then everything falls apart mysteriously.

Service Mesh Hell: We added Istio for "observability" and "security." What we got was 150ms of extra latency because the default health check timeout was 200ms, but our database queries took 300ms. The Istio performance impact is real, and service mesh latency overhead affects every request.

Fix: Tune your service mesh or don't use one. Host networking bypasses all the container networking overhead, but breaks container isolation. Use it for latency-critical services only.

DNS is always broken: Container DNS resolution adds 50-100ms per lookup. Cache DNS aggressively or use IP addresses for internal services. We saw 500ms response times drop to 50ms just by caching DNS for 30 seconds.

Storage: Don't Store Logs in Containers, Ever

This should be obvious, but I've seen it too many times. Logging to files inside containers fills up the container filesystem, Docker starts failing writes, and everything breaks.

Wrong way:

const fs = require('fs');
fs.writeFileSync('/app/logs/error.log', error); // DON'T DO THIS

Right way:

console.error(error); // Log to stdout, ship it elsewhere

Use Docker volumes for databases, not bind mounts. Volumes are 15-20% faster and don't have the permission nightmare of bind mounts.

Real disaster: Our PostgreSQL container filled up its 20GB volume with logs in 3 days. The database went read-only, the app crashed, and I spent Saturday restoring from backup. Now we ship logs to CloudWatch and rotate them daily.

Platform-Specific Pain Points

AWS Graviton: ARM64 is cheaper and faster, but half your Docker images don't support it. AWS Graviton instances are 40% cheaper, but your x86-only images won't run. Build multi-arch images or stick with x86. The Graviton performance benchmarks show significant improvements for most workloads.

Google Cloud Run: Serverless containers sound great until you hit the 15-minute timeout and your batch jobs die halfway through. Use regular GKE for long-running processes.

Azure Container Instances: Cheap for development, expensive for production. We moved to AKS and cut our bill by 60%.

Monitoring That Actually Helps

Forget "comprehensive observability strategies." Here's what you actually need to debug production issues at 3 AM:

1. Memory and CPU usage - kubectl top pods works fine
2. Response times - P95, P99 latencies, not averages
3. Error rates - 500s per minute, not just "some errors occurred"
4. Pod restart count - If pods keep restarting, something's wrong

OpenTelemetry is supposed to fix observability. In reality, it adds 20-50ms latency to every request and makes debugging harder. Start simple: log structured JSON to stdout, ship to ELK or CloudWatch, add alerts when things break.

Pro tip: Set up alerts for when your pods restart more than 3 times in 10 minutes. This catches OOMKills, crashes, and configuration problems before users notice. Use Prometheus CPU monitoring for detailed metrics, resource usage monitoring for Grafana dashboards, and proper alerting strategies to stay sane.

Step 1: Stop Everything and Monitor What's Actually Happening

Skip the "comprehensive baseline assessment." Just run kubectl top pods and look at your AWS bill. If pods are getting OOMKilled or your bill doubled, you have problems.

Start here: Install Prometheus and Grafana on your cluster. Yes, it uses 200-500MB RAM per node. Deal with it. The default kube-state-metrics tells you what's actually using resources. Follow the Prometheus operator guide for easier setup.

Reality check: Kubernetes metrics still kinda suck at telling you why things are slow. You'll spend hours staring at graphs that show high CPU usage without explaining whether it's garbage collection, I/O wait, or just your algorithm being shit.

Real monitoring commands:

## See which pods use the most memory
kubectl top pods --all-namespaces --sort-by=memory

## Check which pods are restarting (usually OOMKilled)  
kubectl get pods --all-namespaces --field-selector=status.phase=Running | grep -v " 0 "

## See actual resource usage vs limits
kubectl describe node | grep -A 5 "Allocated resources"

## Find pods that were OOMKilled recently
kubectl get events --all-namespaces | grep "OOMKilling"

## Check for CPU throttling (the silent killer)
kubectl top pods --containers --all-namespaces

Step 2: Fix Your Bloated Images (This Will Save You Money Immediately)

Don't do phased rollouts. Fix the worst images first - the ones that are 1GB+ for no reason.

The Dockerfile that actually works:

## Build stage - keep all the build crap here
FROM node:18-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production --no-audit --no-fund
RUN npm prune --production

## Runtime - only what you need to run
FROM node:18-alpine AS runtime
RUN addgroup -g 1001 nodejs && adduser -u 1001 -g nodejs -s /bin/sh -D nodejs
WORKDIR /app
COPY --from=build --chown=nodejs:nodejs /app/node_modules ./node_modules
COPY --chown=nodejs:nodejs . .
USER nodejs
EXPOSE 3000
CMD ["node", "server.js"]

What this actually does:

• Cuts your image from like 800MB down to around 150MB
• Startup drops from almost a minute to maybe 8-10 seconds
• Less network transfer = faster deployments
• The USER nodejs prevents your app from running as root

Don't use Alpine unless you're prepared for pain. Debian slim is 50MB bigger but saves you days of debugging DNS and SSL issues.

Step 3: Set Resource Limits That Don't Suck

Everyone copies limits from tutorials without testing. This kills pods and wastes money.

How to actually set limits:

1. Deploy without limits, monitor for at least a week
2. Set memory limit to roughly 2x normal usage (handles traffic spikes)
3. Set CPU request to average usage, limit to maybe 3-4x that
4. Watch for OOMKills and throttling, adjust as needed

Real example from production:

resources:
  requests:
    memory: "200Mi"  # What it normally uses
    cpu: "100m"      # 0.1 CPU cores baseline
  limits:
    memory: "400Mi"  # 2x for traffic spikes
    cpu: "500m"      # Can burst to 0.5 cores

Java apps are special snowflakes: They'll use all memory you give them. Set JVM heap to 75% of container memory limit or the JVM will get killed by Kubernetes. The Java memory management in containers docs explain the details.

Step 4: Network Optimization (AKA Stop Using Service Mesh for Everything)

Service meshes like Istio add 100ms+ latency to every request. If you don't need traffic splitting or mutual TLS, don't use one.

What actually works:

• Use Kubernetes services for internal communication
• Set proper health check timeouts (5+ seconds, not 200ms)
• Cache DNS lookups - container DNS is slow
• Use host networking for latency-critical services
• Implement network policies for security without service mesh overhead

DNS caching for Node.js (saves 50-100ms per request):

const dns = require('dns');
dns.setDefaultResultOrder('ipv4first');

// Enable DNS caching
process.env.NODE_OPTIONS = '--dns-result-order=ipv4first';

Step 5: Autoscaling That Actually Works

HPA (Horizontal Pod Autoscaler) is good. VPA (Vertical Pod Autoscaler) is experimental and breaks things. Cluster autoscaling is essential if you use spot instances.

HPA config that doesn't flap:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: app-hpa
spec:
  minReplicas: 2        # Always have 2 running
  maxReplicas: 20       # Don't go crazy
  targetCPUUtilization: 80  # Scale when CPU hits 80%
  scaleDownPolicy:
    stabilizationWindowSeconds: 300  # Wait 5 min before scaling down

Why these numbers work:

• 80% CPU triggers scaling before users start noticing slowdowns
• 5-minute scale-down window prevents constant flapping
• Min replicas prevent cold start delays during traffic spikes

Step 6: Storage and Logging (Don't Fill Up Your Disks)

Storage rules:

• Use persistent volumes for databases
• Use emptyDir for temporary files
• Never store application logs inside containers

Logging that works:

// Log to stdout only - never to files
console.log(JSON.stringify({
  timestamp: new Date().toISOString(),
  level: 'info',
  message: 'User action completed',
  userId: req.user.id,
  action: 'purchase'
}));

Ship logs to CloudWatch, ELK, or wherever. But never, ever log to files inside containers.

Step 7: Monitoring That Helps at 3 AM

Install Prometheus and Grafana. Set up alerts for:

• Pods restarting more than 3 times in 10 minutes (OOMKills, crashes)
• Response time P95 > 1 second (performance degradation)
• Error rate > 1% (something's broken)
• Memory usage > 85% (about to OOMKill)

Useful Grafana dashboard queries:

## Pod restart rate
rate(kube_pod_container_status_restarts_total[10m]) > 0

## Memory usage percentage  
container_memory_usage_bytes / container_spec_memory_limit_bytes * 100 > 85

## Request latency P95
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 1

The Nuclear Options

If everything is still slow and expensive:

1. Switch to ARM64 instances - 40% cheaper on AWS, but test your images first
2. Use spot instances - 70% cheaper but can disappear randomly
3. Implement pod preemption - Kill low-priority pods for high-priority ones
4. Move to serverless - Cloud Run, Fargate, but watch the timeouts

Reality check: Container optimization is not a six-phase enterprise program. It's monitoring your shit, fixing the obvious problems, and setting up alerts so you know when things break. Use Docker best practices, follow Kubernetes troubleshooting guides, and learn from actual production monitoring instead of consultant frameworks. Everything else is consultant bullshit.

Your Next Move

You just learned how to cut your container images from like 2GB down to around 200MB, drop startup times from minutes to seconds, and stop your pods from getting mysteriously killed. But here's the thing - every production environment is different, and your specific fuckups will be uniquely creative.

The three things that will probably save your ass:

1. Monitor first, optimize second - Don't just guess what's broken, actually measure it
2. Start with the biggest problems - Fix those massive 2GB images before you worry about perfect health checks
3. Set up alerts that actually wake you up - Because production failures sure as hell don't respect your sleep schedule

The container ecosystem changes fast, but the fundamentals stay the same: containers will break in new and exciting ways, your monitoring will miss the important stuff, and someone will always set memory limits wrong. But if you follow the nuclear options in this guide, at least your problems will be boring ones instead of career-ending disasters.

Go fix your containers. Your future self will thank you when you're not debugging OOMKilled pods at 3 AM.