Fix Kubernetes OOMKilled Errors (Before They Ruin Your Weekend)

Your pod just died with exit code 137 again. The logs show Reason: OOMKilled and you're about to double the memory limit and hope for the best. Don't. I've been down this road - it just delays the inevitable crash until your traffic spikes hit.

OOMKilled happens when your container gets too hungry for memory and the Linux kernel kills it. Period. No negotiation, no warnings, just dead. The kernel doesn't care about your business logic or graceful shutdown.

Personal Experience: I once spent 6 hours debugging "OOMKilled" pods that turned out to be hitting the PID limit, not memory. The error message lies sometimes, so don't trust it blindly.

The Two Types of OOM Deaths That Will Fuck Up Your Sleep

Type 1: The Obvious Kill - Pod Dies Screaming

This is the OOMKilled error everyone recognizes. Your pod shows OOMKilled, kubectl logs shows exit code 137, and your restart count is climbing faster than your blood pressure. Check the pod status and container states for confirmation.

## Check for obvious OOMKilled errors (you'll run this 50 times)
kubectl get pods | grep -E "(OOMKilled|Error|137)"
kubectl describe pod <pod-name> | grep -A 5 -B 5 "OOMKilled"
kubectl logs <pod-name> --previous | tail -50
## Pro tip: --previous shows logs from the dead container, not the new one

Type 2: The Invisible Kill - Your App Dies But Kubernetes Doesn't Give a Shit

This is the invisible OOM kill nightmare that makes debugging absolute hell. A child process inside your container gets murdered, but PID 1 stays alive, so Kubernetes thinks everything's peachy. The container runtime has no visibility into this.

I learned this the hard way: Spent 8 hours debugging "healthy" pods that were actually dead inside. Child processes were getting OOM killed while the main process kept running like nothing happened.

Symptoms of invisible kills:

• Application becomes unresponsive randomly
• Error rates spike without pod restarts
• Performance degrades over time
• Memory usage stays flat after spikes

How to detect invisible kills:

## Check kernel logs on the node (requires node access)
## See: https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/
journalctl --utc -k | grep -i "killed process"
journalctl --utc -k | grep -i "out of memory"
## Alternative: dmesg | grep -i "killed process"

## Check for memory pressure on nodes
## Reference: https://kubernetes.io/docs/concepts/scheduling-eviction/node-pressure-eviction/
kubectl describe nodes | grep -A 10 -B 10 "MemoryPressure"
## Also check: kubectl get nodes -o wide

## Monitor memory usage patterns
kubectl top pod <pod-name> --containers

The Memory Debugging Process That Actually Works (Instead of Guessing)

Most teams throw darts at memory limits. Here's how to not be one of those teams:

Step 1: Figure Out If It's Actually OOM (It Usually Isn't What You Think)

## Get the full picture of what actually happened
## Reference: https://kubernetes.io/docs/tasks/debug/debug-application/debug-pods/
kubectl describe pod <pod-name> | grep -A 20 "Last State"
kubectl get events --sort-by='.lastTimestamp' | grep <pod-name>
## Events get rotated every hour, so if you're late to the party, tough shit
## See: https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/

## Check actual vs requested memory usage (spoiler: they're never the same)
## Resource docs: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
kubectl describe pod <pod-name> | grep -A 5 -B 5 "Limits"
kubectl top pod <pod-name>  # Shows current usage, not the spike that killed it
## Requires metrics-server: https://github.com/kubernetes-sigs/metrics-server

What you're actually looking for:

• Terminated with Reason: OOMKilled (the smoking gun)
• Memory usage near limits (but remember, spikes don't show up in kubectl top)
• Recent events about memory pressure (if they haven't rotated away yet)

Step 2: Figure Out What's Actually Eating Your Memory

## Get memory breakdown inside the container (if it's still alive)
## Linux memory docs: https://www.kernel.org/doc/Documentation/filesystems/proc.txt
kubectl exec -it <pod-name> -- cat /proc/meminfo | head -10
kubectl exec -it <pod-name> -- ps aux --sort=-%mem | head -20
## Alternative: kubectl exec -it <pod-name> -- top -o %MEM

## Check for memory-mapped files eating space (you'd be surprised)
## Memory mapping info: https://man7.org/linux/man-pages/man5/proc.5.html
kubectl exec -it <pod-name> -- find /proc/*/maps -exec grep -l "rw-" {} \; 2>/dev/null | wc -l
## See also: kubectl exec -it <pod-name> -- cat /proc/*/smaps | grep -i rss

## Monitor memory over time (you'll get bored after 5 minutes and ctrl+c)
kubectl exec -it <pod-name> -- free -h -s 5

Step 3: Profile Memory Usage Patterns

Different applications have different memory patterns:

Java Applications (The Usual Suspects):

For comprehensive Java memory debugging, check the official JVM documentation and Kubernetes Java memory best practices.

## Check heap usage and GC (prepare for disappointment)
kubectl exec -it <pod-name> -- jstat -gc $(pgrep java) 5s

## Get heap dump for analysis (warning: this will freeze your app for 30+ seconds)
kubectl exec -it <pod-name> -- jcmd $(pgrep java) GC.run_finalization
kubectl exec -it <pod-name> -- jcmd $(pgrep java) VM.memory_info
## Don't run this during peak traffic unless you enjoy angry customers

Node.js Applications (Memory Leak Central):

Node.js memory debugging requires understanding V8 memory management and Node.js performance debugging.

## Enable memory profiling (requires app restart, because of course it does)
kubectl exec -it <pod-name> -- node --inspect=0.0.0.0:9229 /app/index.js &
## Use Chrome DevTools to connect and profile - assuming your network config isn't fucked

Python Applications (Surprise Memory Hogs):

Python memory issues are well-documented in the memory profiling guide and Python memory management docs.

## Install memory profiler (assuming pip still works in your locked-down container)
kubectl exec -it <pod-name> -- python -m memory_profiler your_script.py

## Check for too many objects in memory
kubectl exec -it <pod-name> -- python -c "import gc; print(len(gc.get_objects()))"
## If this number is over 100k, you probably have a leak

Real War Stories From The OOMKilled Trenches

War Story #1: The Database Connection Pool From Hell

Our Spring Boot API kept getting murdered every few hours. Memory limit was 2GB, JVM heap was only 1.5GB, but something else was eating 500MB+. Spent 6 hours looking at heap dumps before I realized the problem wasn't on the heap.

Turns out HikariCP was configured for 50 connections, and each connection was caching 10MB of result sets. 50 × 10MB = 500MB of off-heap memory that didn't show up in any JVM monitoring.

Fix: Cut connection pool to 20 and capped result set cache size. Problem solved in 10 minutes once I found the actual issue.

War Story #2: When Winston Tried to Log Everything to Memory

Node.js app went from 200MB to 2GB+ over 48 hours. No obvious leaks in our code - we checked everything twice. Spent two days profiling before finding the real culprit.

Winston was configured to buffer 50,000+ log entries before flushing. Each log entry was ~1KB. Do the math: 50MB+ just sitting in a buffer, growing forever.

Fix: Set buffer size to 1,000 entries max. Memory usage dropped back to 200MB instantly. Sometimes the simplest fixes are the ones you overlook.

War Story #3: The Invisible ArgoCD Massacre

ArgoCD pods looked healthy but apps kept showing "Out of Sync" randomly. No crashes, no restarts, no obvious issues. Took me a week to figure out what was happening.

Turns out helm processes were getting OOM killed during large deployments, but the main ArgoCD process stayed alive. The repo-server had a 128Mi limit, but helm needed 192Mi for complex charts.

Fix: Bumped memory to 256Mi. Moral of the story: invisible OOM kills are the worst kind of debugging nightmare.

This stuff actually works. I've used these techniques to stop getting paged at 3AM about dead pods. The key is understanding that OOMKilled isn't always what it seems - sometimes you need to dig deeper to find what's really killing your containers.

Basic kubectl top tells you memory is high, but it's a lying piece of shit. It shows current usage, not the spike that killed your pod 30 seconds ago. When you need to find what's actually eating your memory, you need better tools than the built-in garbage.

Deep Memory Analysis with Ephemeral Containers (When Your Cluster Admin Actually Enables Them)

Ephemeral containers are awesome for debugging - assuming your security team allows them (they don't). Stable since K8s 1.25+ and widely supported in modern clusters (1.30+), they let you inject debugging tools without rebuilding images. Check the security implications first.

Reality check: Most corporate environments disable ephemeral containers for "security reasons." If yours does, skip to the next section and cry a little.

## Add a debugging container (if your security policies allow it, lol)
## Reference: https://kubernetes.io/docs/tasks/debug/debug-application/debug-running-pod/
kubectl debug <pod-name> -it --image=nicolaka/netshoot --target=<container-name>

## Once inside the ephemeral container, you have access to:
## - top, htop for real-time process monitoring
## - pmap for memory mapping analysis (see: man pmap)  
## - valgrind for memory leak detection (slow as fuck) - https://valgrind.org/docs/manual/mc-manual.html
## - strace for system call tracing (prepare for spam) - https://man7.org/linux/man-pages/man1/strace.1.html
## - /proc filesystem for detailed memory stats - https://man7.org/linux/man-pages/man5/proc.5.html

Actually useful memory analysis inside ephemeral containers:

## Analyze memory maps for the main process
pmap -x $(pgrep -f "your-app-name") | sort -nk3
## This shows you what's actually using memory, not just heap size

## Track memory allocations over time (you'll get bored after 10 minutes)
while true; do
  echo "$(date): $(cat /proc/meminfo | grep MemAvailable)"
  ps aux --sort=-%mem | head -5
  sleep 30
done

## Find processes with suspicious memory usage
for pid in $(pgrep -f "your-app"); do
  echo "PID $pid memory usage:"
  cat /proc/$pid/status | grep -E "VmPeak|VmSize|VmRSS|VmData"
done

Language-Specific Memory Profiling (Because Every Language Leaks Memory Differently)

Java/JVM Memory Deep Dive (Off-Heap Memory Will Fuck You)

Java memory issues are usually off-heap problems that don't show up in heap dumps. Your heap looks fine, but something else is eating 2GB of RAM. Check the JVM memory model and container memory limits:

## Get complete memory breakdown (assuming you can run ephemeral containers)
## JDK tools: https://docs.oracle.com/javase/8/docs/technotes/tools/unix/jcmd.html
kubectl debug <pod-name> -it --image=openjdk:11 --target=<container-name>

## Inside ephemeral container:
jcmd $(pgrep java) VM.memory_info  # The money shot
jcmd $(pgrep java) GC.class_histogram | head -50
jstat -gc $(pgrep java) 250ms 40  # Watch GC activity

## Off-heap memory analysis (this is where your memory went)
jcmd $(pgrep java) VM.native_memory summary
## If this command fails, your JVM wasn't started with -XX:NativeMemoryTracking=summary

Common Java memory issues that will ruin your day:

References: Oracle JVM Troubleshooting Guide, OpenJDK Memory Guide, JVM Container Best Practices

1. DirectByteBuffer leaks - NIO operations not releasing off-heap buffers (fuck Netty)
2. Metaspace exhaustion - Too many classes loaded (looking at you, Spring Boot with 500 dependencies)
3. Code cache overflow - JIT compiler cache full, performance goes to shit
4. Connection pool bloat - HikariCP holding 50 connections × 10MB each = 500MB gone

Java memory debugging script (that actually works):

#!/bin/bash
## Save this as memory-debug.sh and run inside ephemeral container

JAVA_PID=$(pgrep java)
if [ -z "$JAVA_PID" ]; then
  echo "No Java process found, probably already dead"
  exit 1
fi

echo "=== Java Memory Analysis for PID $JAVA_PID ==="

echo "1. Heap Usage:"
jcmd $JAVA_PID GC.run && jcmd $JAVA_PID VM.memory_info | grep -A 10 "Memory"

echo "2. Top Memory-Consuming Classes (the usual suspects):"
jcmd $JAVA_PID GC.class_histogram | head -20

echo "3. Off-Heap Usage (where your memory actually went):"
jcmd $JAVA_PID VM.native_memory summary | grep -E "Total|Java Heap|Class|Thread"

echo "4. GC Performance (how fucked are you?):"
jstat -gc $JAVA_PID | awk '{print "Eden:", $3"KB", "Old:", $7"KB", "GC Time:", $12"s"}'

Node.js Memory Profiling (Welcome to Callback Hell)

Node.js memory leaks are usually closures holding onto massive objects, event listeners that never get removed, or some asshole loading a 50MB JSON file into memory. See the Node.js memory best practices and V8 memory management:

## Enable memory profiling (requires app restart because Node.js is special)
kubectl patch deployment <deployment-name> -p '{"spec":{"template":{"spec":{"containers":[{"name":"<container-name>","args":["--inspect=0.0.0.0:9229","--max-old-space-size=1024","/app/index.js"]}]}}}}'

## Connect to Node.js inspector (assuming your network doesn't hate you)
kubectl debug <pod-name> -it --image=node:18 --target=<container-name>

## Generate heap snapshot (this will freeze your app for 10+ seconds)
node -e "
const v8 = require('v8');
const fs = require('fs');
const snapshot = v8.writeHeapSnapshot();
console.log('Heap snapshot written to', snapshot);
" 
## Warning: heap snapshots are usually 100MB+ and will crash your laptop

Node.js memory leak detection:

// Add this to your application for memory monitoring
setInterval(() => {
  const used = process.memoryUsage();
  console.log({
    timestamp: new Date().toISOString(),
    rss: Math.round(used.rss / 1024 / 1024) + 'MB',
    heapTotal: Math.round(used.heapTotal / 1024 / 1024) + 'MB', 
    heapUsed: Math.round(used.heapUsed / 1024 / 1024) + 'MB',
    external: Math.round(used.external / 1024 / 1024) + 'MB'
  });
}, 30000);

Python Memory Profiling (Global Interpreter Lock Can't Save You Now)

Python memory issues are usually pandas DataFrames eating all your RAM, circular references that never get garbage collected, or some ML library hoarding memory like a dragon:

## Install memory profiling tools (assuming pip still works in your container)
kubectl debug <pod-name> -it --image=python:3.9 --target=<container-name>

## Inside ephemeral container:
pip install memory-profiler pympler psutil  # This will take forever

## Profile memory usage by line (prepare to wait)
python -m memory_profiler your_script.py

## Find what's hogging memory
python -c "
import gc
import collections
counter = collections.Counter()
for obj in gc.get_objects():
    counter[type(obj).__name__] += 1
print('Top memory hogs:', counter.most_common(20))
"

Detecting Invisible OOM Kills

The most frustrating memory issue is when processes die inside containers without Kubernetes knowing:

Node-Level OOM Detection

## Access worker node (method varies by cluster setup)
kubectl debug node/<node-name> -it --image=busybox

## Inside node debugging container:
## Check kernel logs for OOM kills
journalctl --utc -k --since "1 hour ago" | grep -i "killed process"

## Look for memory cgroup violations
dmesg | grep -i "memory cgroup out of memory"

## Find which pods are causing OOM pressure
find /sys/fs/cgroup/memory -name "memory.oom_control" -exec sh -c '
  for f; do
    if [ -f "$f" ]; then
      echo "Cgroup: $f"
      cat "$f" | grep -E "oom_kill|under_oom"
    fi
  done
' _ {} \;

Monitoring Memory Pressure Indicators

## Check for memory pressure on nodes
kubectl get nodes -o custom-columns=NAME:.metadata.name,MEMORY-PRESSURE:.status.conditions[?(@.type==\"MemoryPressure\")].status

## Get detailed memory statistics
kubectl describe nodes | grep -A 20 "Allocated resources"

## Monitor pod memory usage over time
while true; do
  echo "$(date): Memory usage by pod:"
  kubectl top pods --sort-by=memory | head -10
  sleep 60
done

Memory Leak Detection Patterns

Pattern 1: Gradual Memory Growth

## Monitor memory growth over 24 hours
for i in $(seq 1 144); do  # 144 * 10min = 24 hours
  echo "$(date): $(kubectl top pod <pod-name> | grep <pod-name>)"
  sleep 600  # 10 minutes (this will take forever and probably fail halfway through)
done > memory-growth.log

Pattern 2: Spike-and-Hold Memory Pattern

## Detect sudden memory spikes that don't recover
kubectl top pod <pod-name> --no-headers | while read line; do
  current_mem=$(echo $line | awk '{print $3}' | sed 's/Mi//')
  if [ $current_mem -gt 500 ]; then  # Alert if over 500MB
    echo "ALERT: $(date) - Memory spike detected: ${current_mem}MB"
    kubectl describe pod <pod-name> | grep -A 10 "Events:"
  fi
  sleep 30
done

Pattern 3: Memory Fragmentation Issues

## Check for memory fragmentation (inside ephemeral container)
cat /proc/buddyinfo  # Shows memory fragmentation levels
cat /proc/pagetypeinfo | grep -A 3 "Unmovable"

Production Memory Monitoring Setup

For ongoing OOMKilled prevention, implement comprehensive monitoring:

## Example Prometheus monitoring rules for OOMKilled detection
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: oomkilled-alerts
spec:
  groups:
  - name: memory.rules
    rules:
    - alert: PodOOMKilled
      expr: increase(kube_pod_container_status_restarts_total{reason="OOMKilled"}[5m]) > 0
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} was OOMKilled"
    
    - alert: HighMemoryUsage
      expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.8
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod {{ $labels.pod }} memory usage above 80%"

These advanced profiling techniques help you understand exactly what's consuming memory in your containers. The next section covers how to prevent these memory issues from happening in the first place.

Debugging OOMKilled at 3AM sucks. Getting paged because your payment API is dead costs more than just setting proper limits in the first place. Here's how to build apps that don't randomly die from memory issues.

Real talk: I used to get paged 2-3 times a week for OOM issues. After implementing these strategies, I maybe get paged once a month. Your sleep schedule will thank you.

Stop Guessing Memory Limits (Yes, Everyone Does It)

Most teams set memory limits by throwing darts at a board, doubling when stuff breaks, and hoping for the best. This approach wastes money and still gets you paged when traffic spikes. Here's how to actually do it right:

Memory Sizing That Actually Works

Step 1: Actually Measure Memory Usage (Revolutionary Concept)

## Profile memory under normal load for 7 days (if you have the patience)
## Requires metrics-server: https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/
kubectl top pod <pod-name> --no-headers | while read line; do
  echo \"$(date +%s),$(echo $line | awk '{print $3}' | sed 's/Mi//')\" 
done >> memory-baseline.csv

## Analyze the data (basic math, but most people skip this)
awk -F',' '{sum+=$2; if($2>max) max=$2} END {
  print \"Average memory: \" sum/NR \"MB\"
  print \"Peak memory: \" max \"MB\"  
  print \"Recommended limit: \" max*1.5 \"MB\"}
' memory-baseline.csv
## Spoiler: your peak is always 3x your average

Step 2: Load Test or Cry Later

## Use k6, hey, or whatever load testing tool doesn't suck
## Monitor memory while throwing traffic at your app

## Gradual ramp-up test (because going 0-100 will just break everything)
for rps in 10 50 100 200 500; do
  echo \"Testing $rps RPS - hope your app survives\"
  hey -z 300s -q $rps https://api.github.com/users/octocat &
  sleep 60
  kubectl top pod <pod-name> | tee -a load-test-memory.log
  sleep 60  # Time to panic if memory usage looks bad
done

Step 3: Memory Math That Actually Works

Base Memory = What your app needs just to start (minimum to not immediately crash)
Working Memory = Memory during normal operation (when users aren't being assholes)
Spike Buffer = Extra memory for when everything goes wrong (50-100% more)
Garbage Collection = Language tax (JVM: 25%, Node.js: 30%, Go: 15%, Python: 40%)

Total Limit = (Base + Working + Spike Buffer) × (1 + GC Overhead)
## Then add 20% because your estimates are always wrong

Example Java application sizing (that won't get you paged):

## Base: 200MB (JVM startup - Spring Boot is a memory hog)
## Working: 400MB (application + libraries + the 47 dependencies you didn't know about)
## Spike: 200MB (50% buffer because traffic spikes are real)
## GC: 25% overhead (Java's garbage collection tax)
## Total: (200 + 400 + 200) × 1.25 = 1000MB limit
## Reality: management says \"can't you just use 512MB?\"

Implementing Quality of Service (QoS) Classes

Kubernetes QoS classes determine which pods get killed first during memory pressure. Understanding and using QoS strategically prevents critical workloads from being OOM killed. See the resource management docs for details.

QoS Class Configuration Examples

Guaranteed QoS (Highest Priority):

apiVersion: v1
kind: Pod
metadata:
  name: guaranteed-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: \"1Gi\"
        cpu: \"500m\"
      limits:
        memory: \"1Gi\"    # Same as requests
        cpu: \"500m\"      # Same as requests

Burstable QoS (Medium Priority):

apiVersion: v1
kind: Pod  
metadata:
  name: burstable-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: \"512Mi\"
        cpu: \"250m\" 
      limits:
        memory: \"1Gi\"    # Higher than requests
        cpu: \"1000m\"     # Higher than requests

Strategic QoS Usage:

• Critical services (databases, payment APIs): Guaranteed QoS
• Web applications: Burstable QoS with conservative requests
• Batch jobs, dev environments: BestEffort or high-limit Burstable

Application-Level Memory Management

Garbage Collection Optimization

Java/JVM Applications:

See JVM container best practices and OpenJDK container awareness.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: openjdk:11
        env:
        # Optimize GC for container environments
        - name: JAVA_OPTS
          value: >
            -XX:+UseG1GC
            -XX:MaxGCPauseMillis=200
            -XX:+UnlockExperimentalVMOptions
            -XX:+UseCGroupMemoryLimitForHeap
            -XX:MaxRAMPercentage=75
            -XX:+HeapDumpOnOutOfMemoryError
            -XX:HeapDumpPath=/tmp/heap-dump.hprof
        resources:
          requests:
            memory: \"1Gi\"
          limits:
            memory: \"1.5Gi\"

Node.js Applications:

References: Node.js performance best practices and V8 memory optimization.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nodejs-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: node:18
        command: 
        - node
        - --max-old-space-size=1024  # Set heap limit explicitly
        - --optimize-for-size         # Reduce memory footprint
        - index.js
        resources:
          requests:
            memory: \"512Mi\"
          limits:
            memory: \"1.5Gi\"

Memory-Efficient Coding Practices

Connection Pool Management:

Connection pooling guidance: HikariCP best practices, Database connection patterns.

## Example database connection configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: db-config
data:
  database.properties: |
    # Conservative connection pool settings
    hikari.maximumPoolSize=10    # Not 100
    hikari.minimumIdle=5
    hikari.maxLifetime=600000    # 10 minutes
    hikari.idleTimeout=300000    # 5 minutes
    
    # Enable connection leak detection
    hikari.leakDetectionThreshold=60000  # 1 minute

Caching Strategy Implementation:

Caching references: Redis configuration guide, Memcached tuning, Kubernetes caching patterns.

apiVersion: v1
kind: ConfigMap  
metadata:
  name: cache-config
data:
  redis.conf: |
    # Memory-efficient Redis configuration
    maxmemory 256mb
    maxmemory-policy allkeys-lru
    
    # Reduce memory overhead  
    hash-max-ziplist-entries 512
    hash-max-ziplist-value 64
    list-max-ziplist-size 8

Monitoring and Alerting for Proactive Prevention

## References: https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
## Kubernetes monitoring: https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: memory-prevention-alerts
spec:
  groups:
  - name: memory-alerts
    interval: 30s
    rules:
    
    # Alert when memory usage approaches limit
    - alert: MemoryUsageHigh
      expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.85
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: \"Memory usage high for {{ $labels.pod }}\"
        description: \"Pod {{ $labels.pod }} is using {{ $value | humanizePercentage }} of memory limit\"
    
    # Alert on memory growth rate  
    - alert: MemoryGrowthRate
      expr: rate(container_memory_usage_bytes[30m]) > 10485760  # 10MB/30min growth
      for: 15m
      labels:
        severity: warning
      annotations:
        summary: \"Potential memory leak in {{ $labels.pod }}\"
    
    # Node memory pressure detection
    - alert: NodeMemoryPressure
      expr: kubelet_node_condition{condition=\"MemoryPressure\", status=\"true\"} == 1
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: \"Node {{ $labels.node }} under memory pressure\"

Vertical Pod Autoscaler (VPA) Configuration

## VPA documentation: https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler
## Installation guide: https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#installation
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: memory-optimization-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: your-app
  updatePolicy:
    updateMode: \"Auto\"  # Or \"Off\" for recommendations only
  resourcePolicy:
    containerPolicies:
    - containerName: app
      maxAllowed:
        memory: \"8Gi\"    # Prevent runaway scaling
      minAllowed:  
        memory: \"256Mi\"  # Minimum viable memory
      controlledResources: [\"memory\"]

Cluster-Level Memory Management

Resource Quotas for Namespace-Level Control

## Resource quota docs: https://kubernetes.io/docs/concepts/policy/resource-quotas/
## Limit ranges: https://kubernetes.io/docs/concepts/policy/limit-range/
apiVersion: v1
kind: ResourceQuota
metadata:
  name: memory-quota
  namespace: production
spec:
  hard:
    requests.memory: \"100Gi\"    # Total memory requests
    limits.memory: \"200Gi\"      # Total memory limits
    pods: \"50\"                  # Maximum pods
    
## Ensure every pod has resource limits
---
apiVersion: v1
kind: LimitRange  
metadata:
  name: memory-limit-range
  namespace: production
spec:
  limits:
  - default:        # Default limits
      memory: \"1Gi\"
    defaultRequest: # Default requests  
      memory: \"512Mi\"
    max:           # Maximum allowed
      memory: \"8Gi\"
    min:           # Minimum required
      memory: \"128Mi\"
    type: Container

Node Selector and Affinity for Memory-Intensive Workloads

## Node affinity docs: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/
## Resource management: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
apiVersion: apps/v1
kind: Deployment
metadata:
  name: memory-intensive-app
spec:
  template:
    spec:
      # Schedule on high-memory nodes
      nodeSelector:
        node-type: \"memory-optimized\"
      
      # Avoid co-location with other memory-heavy pods  
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                type: \"memory-intensive\" 
            topologyKey: kubernetes.io/hostname
      
      containers:
      - name: app
        image: memory-heavy-app:latest
        resources:
          requests:
            memory: \"4Gi\"
          limits:
            memory: \"8Gi\"

Emergency Response Procedures

When prevention fails, have automated responses ready:

Automatic Pod Cycling During High Memory

apiVersion: v1
kind: ConfigMap
metadata:
  name: memory-monitor-script
data:
  monitor.sh: |
    #!/bin/bash
    THRESHOLD=85  # Memory usage percentage threshold
    
    while true; do
      kubectl top pods --no-headers | while read pod_line; do
        POD=$(echo $pod_line | awk '{print $1}')
        MEMORY_PERCENT=$(kubectl top pod $POD --no-headers | awk '{print $3}' | sed 's/%//')
        
        if [ $MEMORY_PERCENT -gt $THRESHOLD ]; then
          echo \"WARNING: $POD memory at ${MEMORY_PERCENT}% - cycling pod\"
          kubectl delete pod $POD
          sleep 30  # Allow time for new pod to start
        fi
      done
      sleep 300  # Check every 5 minutes
    done

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: memory-monitor
spec:
  schedule: \"*/5 * * * *\"  # Every 5 minutes
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: monitor
            image: bitnami/kubectl:latest
            command: [\"bash\", \"/scripts/monitor.sh\"]
            volumeMounts:
            - name: script
              mountPath: /scripts
          volumes:
          - name: script
            configMap:
              name: memory-monitor-script
          restartPolicy: OnFailure

These prevention strategies work together to create a memory-resilient Kubernetes environment. The key is implementing multiple layers: proper sizing, QoS configuration, application optimization, monitoring, and automated responses. This comprehensive approach prevents most OOMKilled errors before they impact production.