Zig Memory Debugging: Production Issues & Prevention

Critical Failure Modes

Container OOMKiller Deaths

• Symptom: App dies with OutOfMemory despite host showing 20GB+ available memory
• Root Cause: Container limits (512MB) enforced while app thinks it has full system RAM access
• Detection: cat /sys/fs/cgroup/memory/memory.limit_in_bytes shows actual limit
• Impact: Instant termination without warning or graceful shutdown
• Quick Fix: Double container memory allocation as temporary measure
• Real Fix: Process files in 64KB chunks instead of loading entirely into memory

Memory Leaks in Production

• Pattern: App starts at 100MB, grows to 2GB over hours/days, dies during off-hours
• Primary Cause: ArenaAllocator in request handlers without arena.reset(.retain_capacity)
• Detection Threshold: >1MB/hour growth indicates leak in stable services
• Early Warning: Alert at 85% container memory gives ~30 minutes before OOMKiller
• Common Sources: ArenaAllocator accumulation, missing defer cleanup in error paths

Use-After-Free in Production

• Why Hidden in Development: DebugAllocator never reuses memory addresses
• Production Exposure: Fast allocators aggressively recycle memory addresses
• Manifestation: Segfaults when freed pointers reference new data
• Detection Strategy: Enable core dumps with ulimit -c unlimited

Allocator Performance vs Safety Trade-offs

Allocator	Speed	Debug Info	Production Use	Memory Overhead
DebugAllocator	500ms vs 100ms	Extensive	NO - Too slow	High
SmpAllocator	Fast	Minimal	YES	Low
ArenaAllocator	Fast	None	YES with reset	Medium
page_allocator	Fastest	OS-level only	YES	Minimal

Critical Decision Point: DebugAllocator catches every leak but makes APIs timeout (100ms → 500ms response times)

Production Memory Debugging Strategies

When Stack Traces Are Useless

# Emergency evidence collection (before restart)
ps aux > memory-snapshot.txt
cat /proc/meminfo > system-memory.txt
dmesg | grep -i "killed process" >> oom-killer.log

Memory Growth Detection

# Continuous monitoring for leak detection
while true; do
    RSS=$(ps -o rss= -p $(pgrep your-service))
    echo "$(date '+%H:%M:%S') RSS: ${RSS}KB" | tee -a memory.log
    sleep 30
done

Container Memory Investigation

# Check actual vs expected limits
cat /sys/fs/cgroup/memory/memory.limit_in_bytes  # Actual container limit
cat /sys/fs/cgroup/memory/memory.usage_in_bytes  # Current usage
ps -o rss= -p $(pgrep your-app)                 # Process RSS

Common Code Patterns That Cause Production Failures

ArenaAllocator Memory Leak

// WRONG - Accumulates memory forever
var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit(); // Only runs when server shuts down

while (handleRequest()) { // Infinite loop
    const response = try arena.allocator().alloc(u8, request.size);
    processRequest(response);
    // Memory accumulates here forever
}

// CORRECT - Reset arena after each request
var arena = std.heap.ArenaAllocator.init(allocator);
defer arena.deinit();

while (handleRequest()) {
    defer arena.reset(.retain_capacity); // Critical line
    const response = try arena.allocator().alloc(u8, request.size);
    processRequest(response);
}

OutOfMemory Crash Prevention

// WRONG - Crashes entire service on large upload
const file_content = try std.fs.cwd().readFileAlloc(allocator, path, std.math.maxInt(usize));

// CORRECT - Graceful degradation
return allocator.alloc(u8, size) catch |err| switch (err) {
    error.OutOfMemory => {
        std.log.err("Upload too large: {} bytes", .{size});
        return error.RequestTooLarge; // Return HTTP 413 instead of crashing
    },
    else => return err,
};

File Processing Without Memory Explosion

// WRONG - Loads entire file into memory
const file_content = try std.fs.cwd().readFileAlloc(allocator, path, std.math.maxInt(usize));

// CORRECT - Process in chunks
var file = try std.fs.cwd().openFile(path, .{});
defer file.close();
var reader = file.reader();

const chunk_size = 64 * 1024; // 64KB chunks
var buffer: [chunk_size]u8 = undefined;
while (try reader.readAll(buffer[0..])) |bytes_read| {
    if (bytes_read == 0) break;
    try processChunk(buffer[0..bytes_read]);
}

Production Memory Monitoring Implementation

Essential Metrics

• RSS growth rate (MB/hour) - not just current usage
• Container memory utilization (percentage of limit)
• Large allocation frequency (>10MB requests)
• Allocation failure rate if supported by allocator

Early Warning Thresholds

• Memory growth >1MB/hour in stable services
• Memory usage >85% of container limit (gives ~30min before OOMKiller)
• Peak request memory >2x normal baseline
• Allocation failures >0.1% of requests

Production Memory Tracker

const ProductionTracker = struct {
    child: std.mem.Allocator,
    total_allocations: std.atomic.Atomic(usize),
    total_deallocations: std.atomic.Atomic(usize),
    bytes_allocated: std.atomic.Atomic(usize),

    pub fn getOutstandingAllocations(self: *ProductionTracker) usize {
        const allocs = self.total_allocations.load(.monotonic);
        const frees = self.total_deallocations.load(.monotonic);
        return allocs - frees;
    }
};

Container Memory Configuration

Production Memory Strategy

• Safety Margin: Container limit should be 20-30% higher than peak tested usage
• Reserve Memory: Account for OS operations and monitoring tools
• Fragmentation Buffer: Long-running processes need extra headroom

Memory Pressure Response

pub fn handleMemoryPressure(current_usage: usize, limit: usize) void {
    const usage_percent = (current_usage * 100) / limit;

    if (usage_percent > 85) {
        // Start rejecting non-critical requests
        setRequestRejectionThreshold(0.1);
        std.log.warn("Memory pressure at {}%, implementing restrictions", .{usage_percent});
    }

    if (usage_percent > 95) {
        // Emergency response - reject all but essential requests
        setRequestRejectionThreshold(0.9);
        std.log.err("Critical memory pressure at {}%", .{usage_percent});
    }
}

Outage Response Procedures

Immediate Stabilization (When Production is On Fire)

1. Don't restart yet - grab evidence first
2. Band-aid the bleeding: Double container memory temporarily
3. Rate limit requests to reduce memory pressure
4. Route traffic away from broken instances
5. Monitor for recurrence with watch -n 10 'ps -o rss $(pgrep your-service)'

Root Cause Analysis Checklist

• Large file processing operations that scale with input size
• Batch operations without intermediate cleanup
• API endpoints triggering complex data transformations
• Background tasks with synchronization issues

Testing and Prevention

CI/CD Memory Testing

# Memory leak detection in CI
zig test --test-filter "*memory*" -fno-strip -fno-sanitize-c

# Load testing with memory constraints
ulimit -v 1048576  # Limit virtual memory to 1GB
timeout 300 ./run-load-test

# Memory regression detection
CURRENT_PEAK=$(./measure-peak-memory)
if [ $CURRENT_PEAK -gt $((BASELINE_PEAK + 10485760)) ]; then  # 10MB increase
    echo "Memory regression detected"
    exit 1
fi

Memory-Specific Load Testing Scenarios

1. Peak Memory Pressure: Gradually increase concurrent requests until allocation failures
2. Large Request Testing: Test with realistic maximum request sizes
3. Long-Running Stability: 24-48 hours under moderate load
4. Memory Recovery: Verify memory returns to baseline after load spikes

Critical Configuration Settings

Container Memory Limits

# Docker memory configuration
docker run -m 1g your-app

# Kubernetes resource limits
resources:
  limits:
    memory: "1Gi"

Core Dump Collection

# Enable core dumps for crash analysis
ulimit -c unlimited
echo '/var/cores/core.%e.%p.%t' > /proc/sys/kernel/core_pattern

Memory Monitoring Script

#!/bin/bash
while true; do
    MEMORY_USAGE=$(cat /sys/fs/cgroup/memory/memory.usage_in_bytes)
    MEMORY_LIMIT=$(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)
    PERCENT=$((MEMORY_USAGE * 100 / MEMORY_LIMIT))

    if [ $PERCENT -gt 90 ]; then
        echo "WARNING: High memory usage detected: ${PERCENT}%"
    fi

    sleep 60
done

When to Use Different Allocators

Use Case	Recommended Allocator	Reason
Web API requests	ArenaAllocator with reset	Bulk cleanup after request
Long-running services	SmpAllocator	Thread-safe, production performance
Development/Testing	DebugAllocator	Leak detection and debugging
Large file processing	Streaming with small buffers	Avoid loading entire files
Real-time systems	Custom allocator	Deterministic allocation times

Performance Impact Data

• DebugAllocator: 100ms → 500ms response times (5x slower)
• Memory leak detection: Fails at >1MB/hour growth in stable services
• Container overhead: 20-30% safety margin needed above peak usage
• OOMKiller warning time: ~30 minutes at 85% container memory
• Core dump analysis: Essential when stack traces point to malloc()

Breaking Points and Failure Modes

• 1000+ spans: UI debugging becomes impossible
• 512MB container limit: Common misconfiguration causing OOM with abundant host memory
• No swap: Kernel more aggressive about killing processes
• Memory fragmentation: Long-running services need extra headroom
• Use-after-free: Only surfaces when production allocators recycle addresses

This technical reference provides complete operational intelligence for implementing, debugging, and preventing memory issues in production Zig applications.