The 3AM Production Debugging Questions

Q

Why does my Node.js app randomly die with "JavaScript heap out of memory"?

A

You've got a memory leak. Something is holding onto objects that should be garbage collected. Common culprits: global arrays that keep growing, event listeners that never get removed, timers that never get cleared, or database connections that never close. The heap fills up until V8 can't allocate more memory.

Q

How do I tell if it's a memory leak or just high memory usage?

A

Run htop or check your monitoring. Real memory leaks show steadily increasing RSS over time, even when request volume is stable. Normal high usage spikes up during traffic and drops back down. If your baseline memory usage creeps up every day, that's a leak.

Q

What's this "ECONNREFUSED 127.0.0.1:3000" error that appears randomly?

A

Your app crashed and restarted. The error happens because something tries to connect to your Node process while it's dead. Check your PM2 logs with pm2 logs

  • you'll probably see an uncaught exception or out of memory error right before the ECONNREFUSED.
Q

My app uses 100% CPU even when there are no requests. What's wrong?

A

Event loop is blocked. Something is running synchronous code that never yields. Usually JSON.parse() on huge objects, regex on massive strings, or crypto operations on the main thread. Use node --prof app.js to profile and find the hot path.

Q

How do I debug memory leaks in production without taking down the server?

A

Generate heap snapshots: kill -USR2 <pid> if you have heapdump installed, or use Chrome DevTools with --inspect. Compare snapshots taken 10 minutes apart to see what objects are growing. Don't use --inspect in production unless you're actively debugging

  • it's a security risk.

The Memory Leak Patterns That Will Kill Your App

Memory Leak Prevention

Pattern #1: The Global Array That Never Stops Growing

This is the classic newbie mistake that takes down production apps:

// This will eventually crash your app
const logs = [];
app.post('/api/log', (req, res) => {
    logs.push({ message: req.body.message, timestamp: Date.now() });
    res.json({ success: true });
});

Why it breaks: That logs array lives forever and grows with every request. After 50,000 log entries, you're using hundreds of megabytes just for logs. After a million, you're dead.

The fix: Add limits and rotation.

const MAX_LOGS = 1000;
const logs = [];

app.post('/api/log', (req, res) => {
    logs.push({ message: req.body.message, timestamp: Date.now() });
    
    // Keep only the latest 1000 entries
    if (logs.length > MAX_LOGS) {
        logs.splice(0, logs.length - MAX_LOGS);
    }
    
    res.json({ success: true });
});

I've seen this exact pattern kill three different production apps. The logs array consumed 2GB of RAM before the app crashed with "heap out of memory".

Pattern #2: Event Listeners That Live Forever

Node.js event emitters leak memory when you keep adding listeners without removing them:

// Memory leak waiting to happen
const { EventEmitter } = require('events');
const emitter = new EventEmitter();

function attachUser(userId) {
    emitter.on('notification', (data) => {
        sendToUser(userId, data);
    });
}

// Every user connection adds another listener
// After 10,000 users, you have 10,000 listeners

The symptoms: Your app starts normally but gets slower over time. Eventually Node.js warns about memory leaks:

(node:1234) MaxListenersExceededWarning: Possible EventEmitter memory leak detected.
11 notification listeners added. Use emitter.setMaxListeners() to increase limit

The fix: Remove listeners when you're done with them.

function attachUser(userId) {
    const handler = (data) => sendToUser(userId, data);
    
    emitter.on('notification', handler);
    
    // Return cleanup function
    return () => emitter.removeListener('notification', handler);
}

// Usage: store the cleanup function and call it when user disconnects
const cleanup = attachUser('user123');
// Later...
cleanup();

Pattern #3: Database Connections Left Open

This one killed my app on Black Friday. Heavy traffic + connection leaks = database refusing new connections:

// DON'T DO THIS - leaks database connections
async function getUser(id) {
    const client = await pool.connect();
    const result = await client.query('SELECT * FROM users WHERE id = $1', [id]);
    // Forgot to release the connection
    return result.rows[0];
}

After 100 requests, your connection pool is exhausted. New requests hang forever waiting for a connection that never comes. This is documented in the PostgreSQL pooling guide and affects MySQL connections too.

The fix: Always release in a finally block.

async function getUser(id) {
    const client = await pool.connect();
    try {
        const result = await client.query('SELECT * FROM users WHERE id = $1', [id]);
        return result.rows[0];
    } finally {
        client.release(); // Always executes
    }
}

Pro tip: Use pg-pool or similar libraries that automatically return connections on query completion. This pattern is recommended in the Node.js database best practices and Sequelize documentation:

// This handles connection management for you
const result = await pool.query('SELECT * FROM users WHERE id = $1', [id]);

Pattern #4: Timers That Never Die

Setters and intervals that outlive their usefulness:

// Creates a timer for every user session
function startUserSession(sessionId) {
    const timer = setInterval(() => {
        pingUser(sessionId);
    }, 30000);
    
    // Timer keeps running even after user disconnects
    sessions[sessionId] = { id: sessionId, timer };
}

The fix: Clear timers when sessions end.

function startUserSession(sessionId) {
    const timer = setInterval(() => {
        pingUser(sessionId);
    }, 30000);
    
    sessions[sessionId] = { 
        id: sessionId, 
        timer,
        cleanup: () => clearInterval(timer)
    };
    
    return sessions[sessionId];
}

function endUserSession(sessionId) {
    const session = sessions[sessionId];
    if (session) {
        session.cleanup(); // Clear the timer
        delete sessions[sessionId];
    }
}

Debugging Tools That Actually Work

Option 1: Chrome DevTools (Best for development)

Start your app with debugging enabled according to the official Node.js debugging guide:

node --inspect app.js

Open Chrome and go to chrome://inspect. Click "inspect" next to your Node process as documented in Chrome DevTools memory profiling.

Go to the Memory tab, take a heap snapshot, run some traffic, take another snapshot. Compare to see what's growing using the memory analysis techniques.

Option 2: Clinic.js (Best for production profiling)

Clinic.js Performance Tool

Install clinic.js: npm install -g clinic

## Profile your app under load
clinic doctor -- node app.js

## Generate load in another terminal
## Then stop clinic with Ctrl+C

## Open the HTML report

Clinic shows you memory usage over time, event loop lag, and CPU usage. The flame graph highlights exactly which functions are using the most memory.

Option 3: Automatic heap dumps on crashes

Add this to your app to generate heap dumps when memory runs low:

const v8 = require('v8');
const fs = require('fs');

// Generate heap dump when memory usage gets high
setInterval(() => {
    const usage = process.memoryUsage();
    const heapUsedMB = usage.heapUsed / 1024 / 1024;
    
    if (heapUsedMB > 400) { // Adjust threshold as needed
        const filename = `heap-${Date.now()}.heapsnapshot`;
        const heapSnapshot = v8.writeHeapSnapshot(filename);
        console.log(`Heap dump written to ${heapSnapshot}`);
    }
}, 60000);

Memory Leak Prevention in Code Reviews

Look for these red flags in pull requests:

  • Global arrays or objects that grow over time
  • Event listeners added without corresponding removal
  • Database queries without proper connection handling
  • Timers/intervals without cleanup
  • Large objects held in closures during async operations
  • Caches without size limits or TTL

The golden rule: For every resource you allocate, have a plan to clean it up.

Memory leaks aren't magic - they're just references that stick around longer than they should. Master these patterns and you'll write Node.js code that runs for months without issues.

Advanced Debugging Scenarios (The Nasty Ones)

Q

My heap snapshots show DOM elements are leaking, but this is a server app with no DOM?

A

You're probably using jsdom or puppeteer for testing or server-side rendering and not cleaning up properly. Each jsdom instance creates hundreds of DOM objects. Call window.close() or browser.close() in your cleanup code.

Q

The heap dump shows thousands of "system / Context" objects. What are these?

A

Usually VM contexts from running untrusted JavaScript code. If you're using vm.runInNewContext() or similar, each execution creates a context that doesn't get garbage collected immediately. Switch to vm.runInThisContext() or add explicit cleanup.

Q

My app leaks memory only under high load, but not during normal traffic?

A

Race conditions in your cleanup code. During high load, requests might finish in different order than expected, causing some cleanup functions to be skipped. Add defensive checks: if (resource && resource.cleanup) resource.cleanup().

Q

The memory keeps growing but heap snapshots show nothing suspicious?

A

It's not a JavaScript heap leak - it's native memory or buffers. Check for:

  • Image processing libraries that don't free native memory
  • Crypto operations that allocate native buffers
  • File handles left open (check with lsof -p <pid>)
  • Native modules with their own memory management issues

Use process.memoryUsage() to see the difference between heapUsed (JavaScript objects) and rss (total memory). If rss grows but heapUsed stays flat, it's native memory.

Q

My Node.js app works fine for hours, then suddenly spikes to 100% CPU and becomes unresponsive?

A

Event loop blocking from a runaway regular expression. The infamous "ReDoS" (Regular Expression Denial of Service). Check your regex patterns for exponential backtracking:

// This regex can take forever on certain inputs
const badRegex = /^(a+)+$/;
const input = 'aaaaaaaaaaaaaaaaaaaaaaaX';
console.time('test');
badRegex.test(input); // Might run for minutes
console.timeEnd('test');

Fix: Use safe-regex to detect dangerous patterns, or add timeouts around regex operations.

Q

How do I debug "socket hang up" errors that happen randomly?

A

Enable request/response logging to see the full HTTP conversation:

app.use((req, res, next) => {
    console.log(`${req.method} ${req.url} - Request started`);
    
    res.on('close', () => {
        console.log(`${req.method} ${req.url} - Response closed`);
    });
    
    res.on('finish', () => {
        console.log(`${req.method} ${req.url} - Response finished`);
    });
    
    next();
});

"Socket hang up" usually means the client disconnected before your response finished. If it happens frequently, check for:

  • Slow database queries that timeout
  • Large responses that take too long to send
  • Client-side timeout settings that are too aggressive

Production Debugging War Stories and Tools

Performance Monitoring

The Buffer Leak That Cost Us $10K

We had a Node.js API that processed uploaded images. Everything worked fine until one day our AWS bill spiked - memory usage went from 512MB to 4GB per container.

The culprit was this innocent-looking code:

app.post('/upload', upload.single('image'), (req, res) => {
    const buffer = fs.readFileSync(req.file.path);
    
    // Process image...
    const processed = sharp(buffer).resize(800, 600).jpeg().toBuffer();
    
    res.json({ success: true, size: processed.length });
    // buffer and processed never get cleaned up
});

The problem: Large Buffer objects (5-10MB each) were staying in memory long after the request finished. Node's garbage collector is lazy about cleaning up large objects, so they accumulated. This is a common issue documented in Sharp's memory usage guide and Multer file handling best practices. The Node.js Buffer documentation explains the underlying memory pooling behavior.

The fix: Explicit cleanup and streaming instead of loading everything into memory:

app.post('/upload', upload.single('image'), async (req, res) => {
    try {
        const transform = sharp()
            .resize(800, 600)
            .jpeg();
            
        const stream = fs.createReadStream(req.file.path)
            .pipe(transform);
            
        const chunks = [];
        for await (const chunk of stream) {
            chunks.push(chunk);
        }
        
        const result = Buffer.concat(chunks);
        res.json({ success: true, size: result.length });
        
    } finally {
        // Clean up temp file immediately
        fs.unlinkSync(req.file.path);
    }
});

Lesson learned: Monitor memory usage per request type. Image processing, PDF generation, and file uploads are common sources of memory spikes. Check out memory management best practices and stream processing patterns for handling large files efficiently.

The Express Session Store That Killed Production

Our login system used the default in-memory session store. Worked great with 100 users. Died horribly with 10,000.

Every user session was stored in a JavaScript object that never expired:

// Default Express session configuration
app.use(session({
    secret: 'keyboard cat',
    // No store specified = in-memory storage
    // No maxAge = sessions never expire
}));

After running for a week, the sessions object had 50,000 entries consuming 2GB of RAM.

The fix: External session storage with automatic expiration:

const RedisStore = require('connect-redis')(session);
const redis = require('redis');
const client = redis.createClient();

app.use(session({
    store: new RedisStore({ client: client }),
    secret: 'keyboard cat',
    cookie: { maxAge: 24 * 60 * 60 * 1000 }, // 24 hours
    resave: false,
    saveUninitialized: false
}));

Moral: Never use in-memory session storage in production. Use Redis, Memcached, or a database. The express-session documentation lists all compatible session stores. Consider connect-redis or connect-mongo for popular options.

Tools for Debugging Memory Issues in Production

1. Process Memory Monitoring

Add this to your app to track memory trends:

const memoryStats = {
    samples: [],
    maxSamples: 100
};

setInterval(() => {
    const usage = process.memoryUsage();
    const sample = {
        timestamp: Date.now(),
        rss: Math.round(usage.rss / 1024 / 1024), // MB
        heapUsed: Math.round(usage.heapUsed / 1024 / 1024), // MB
        heapTotal: Math.round(usage.heapTotal / 1024 / 1024), // MB
        external: Math.round(usage.external / 1024 / 1024) // MB
    };
    
    memoryStats.samples.push(sample);
    if (memoryStats.samples.length > memoryStats.maxSamples) {
        memoryStats.samples.shift();
    }
    
    // Warn if memory usage is trending upward
    if (memoryStats.samples.length >= 10) {
        const recent = memoryStats.samples.slice(-10);
        const trend = recent[9].heapUsed - recent[0].heapUsed;
        
        if (trend > 50) { // Growing by 50MB over 10 samples
            console.warn(`Memory trending upward: +${trend}MB over last 10 samples`);
        }
    }
}, 30000);

// Expose memory stats endpoint for monitoring
app.get('/debug/memory', (req, res) => {
    res.json(memoryStats);
});

2. CPU Profiling for Event Loop Issues

When your app becomes unresponsive, generate a CPU profile:

## Start profiling
node --prof app.js

## Run your workload for 30-60 seconds
## Stop the app (Ctrl+C)

## Generate human-readable profile
node --prof-process isolate-0x*.log > profile.txt

Look for functions that consume high percentages of CPU time. Usually it's:

  • JSON.parse/stringify on huge objects
  • Regular expressions with catastrophic backtracking
  • Synchronous crypto operations
  • Nested loops processing large arrays

3. Heap Dump Analysis

For memory leaks that are hard to reproduce:

const v8 = require('v8');

// Expose heap dump endpoint (only enable in staging)
if (process.env.NODE_ENV === 'staging') {
    app.get('/debug/heapdump', (req, res) => {
        const filename = `heap-${Date.now()}.heapsnapshot`;
        v8.writeHeapSnapshot(filename);
        res.json({ file: filename, message: 'Heap dump saved' });
    });
}

Download the .heapsnapshot file and open it in Chrome DevTools. Compare dumps taken before and after running your suspect operations. The V8 heap snapshot guide explains how to interpret the results.

4. Automatic Monitoring with Built-in Metrics

Track key performance indicators:

const metrics = {
    requestCount: 0,
    errorCount: 0,
    responseTimeSum: 0,
    maxMemory: 0
};

app.use((req, res, next) => {
    const startTime = Date.now();
    metrics.requestCount++;
    
    res.on('finish', () => {
        const duration = Date.now() - startTime;
        metrics.responseTimeSum += duration;
        
        if (res.statusCode >= 400) {
            metrics.errorCount++;
        }
        
        const memUsage = process.memoryUsage().heapUsed;
        if (memUsage > metrics.maxMemory) {
            metrics.maxMemory = memUsage;
        }
    });
    
    next();
});

// Reset and log metrics every minute
setInterval(() => {
    const avgResponseTime = metrics.responseTimeSum / metrics.requestCount || 0;
    const errorRate = (metrics.errorCount / metrics.requestCount) * 100 || 0;
    
    console.log(JSON.stringify({
        timestamp: new Date().toISOString(),
        requests: metrics.requestCount,
        avgResponseMs: Math.round(avgResponseTime),
        errorRate: errorRate.toFixed(2) + '%',
        maxMemoryMB: Math.round(metrics.maxMemory / 1024 / 1024),
        currentMemoryMB: Math.round(process.memoryUsage().heapUsed / 1024 / 1024)
    }));
    
    // Reset counters
    Object.keys(metrics).forEach(key => {
        if (key !== 'maxMemory') metrics[key] = 0;
    });
}, 60000);

The Nuclear Option: Restarting Leaked Processes

Sometimes the leak is too subtle to find quickly and production is suffering. Use PM2's memory monitoring to automatically restart leaky processes:

// ecosystem.config.js
module.exports = {
  apps: [{
    name: 'api',
    script: './app.js',
    instances: 'max',
    max_memory_restart: '500M', // Restart if memory exceeds 500MB
    env_production: {
      NODE_ENV: 'production'
    }
  }]
};

This is a band-aid, not a fix. But it keeps your app running while you debug the root cause.

Remember: Debugging production issues is about having the right telemetry in place before problems happen. Instrument early, monitor constantly, and always have a rollback plan. Learn more about Node.js observability patterns and production monitoring strategies.

Essential Node.js Debugging Resources

Related Tools & Recommendations

tool
Similar content

React Production Debugging: Fix App Crashes & White Screens

Five ways React apps crash in production that'll make you question your life choices.

React
/tool/react/debugging-production-issues
100%
tool
Similar content

Node.js Performance Optimization: Boost App Speed & Scale

Master Node.js performance optimization techniques. Learn to speed up your V8 engine, effectively use clustering & worker threads, and scale your applications e

Node.js
/tool/node.js/performance-optimization
97%
tool
Similar content

Webpack: The Build Tool You'll Love to Hate & Still Use in 2025

Explore Webpack, the JavaScript build tool. Understand its powerful features, module system, and why it remains a core part of modern web development workflows.

Webpack
/tool/webpack/overview
97%
tool
Similar content

Node.js Overview: JavaScript Runtime, Production Tips & FAQs

Explore Node.js: understand this powerful JavaScript runtime, learn essential production best practices, and get answers to common questions about its performan

Node.js
/tool/node.js/overview
95%
tool
Similar content

Express.js - The Web Framework Nobody Wants to Replace

It's ugly, old, and everyone still uses it

Express.js
/tool/express/overview
95%
tool
Similar content

Node.js Microservices: Avoid Pitfalls & Build Robust Systems

Learn why Node.js microservices projects often fail and discover practical strategies to build robust, scalable distributed systems. Avoid common pitfalls and e

Node.js
/tool/node.js/microservices-architecture
92%
tool
Similar content

Node.js Security Hardening Guide: Protect Your Apps

Master Node.js security hardening. Learn to manage npm dependencies, fix vulnerabilities, implement secure authentication, HTTPS, and input validation.

Node.js
/tool/node.js/security-hardening
92%
tool
Similar content

Node.js Production Deployment - How to Not Get Paged at 3AM

Optimize Node.js production deployment to prevent outages. Learn common pitfalls, PM2 clustering, troubleshooting FAQs, and effective monitoring for robust Node

Node.js
/tool/node.js/production-deployment
89%
troubleshoot
Similar content

Fix TypeScript Module Resolution Errors: Stop 'Cannot Find Module'

Stop wasting hours on "Cannot find module" errors when everything looks fine

TypeScript
/troubleshoot/typescript-module-resolution-error/module-resolution-errors
84%
tool
Similar content

Debugging Broken Truffle Projects: Emergency Fix Guide

Debugging Broken Truffle Projects - Emergency Guide

Truffle Suite
/tool/truffle/debugging-broken-projects
84%
tool
Similar content

GraphQL Overview: Why It Exists, Features & Tools Explained

Get exactly the data you need without 15 API calls and 90% useless JSON

GraphQL
/tool/graphql/overview
79%
tool
Similar content

Node.js Docker Containerization: Setup, Optimization & Production Guide

Master Node.js Docker containerization with this comprehensive guide. Learn why Docker matters, optimize your builds, and implement advanced patterns for robust

Node.js
/tool/node.js/docker-containerization
79%
tool
Similar content

Docker: Package Code, Run Anywhere - Fix 'Works on My Machine'

No more "works on my machine" excuses. Docker packages your app with everything it needs so it runs the same on your laptop, staging, and prod.

Docker Engine
/tool/docker/overview
76%
tool
Similar content

npm - The Package Manager Everyone Uses But Nobody Really Likes

It's slow, it breaks randomly, but it comes with Node.js so here we are

npm
/tool/npm/overview
76%
troubleshoot
Similar content

Fix MongoDB "Topology Was Destroyed" Connection Pool Errors

Production-tested solutions for MongoDB topology errors that break Node.js apps and kill database connections

MongoDB
/troubleshoot/mongodb-topology-closed/connection-pool-exhaustion-solutions
76%
tool
Similar content

npm Enterprise Troubleshooting: Fix Corporate IT & Dev Problems

Production failures, proxy hell, and the CI/CD problems that actually cost money

npm
/tool/npm/enterprise-troubleshooting
76%
integration
Similar content

MongoDB Express Mongoose Production: Deployment & Troubleshooting

Deploy Without Breaking Everything (Again)

MongoDB
/integration/mongodb-express-mongoose/production-deployment-guide
71%
howto
Similar content

Install Node.js & NVM on Mac M1/M2/M3: A Complete Guide

My M1 Mac setup broke at 2am before a deployment. Here's how I fixed it so you don't have to suffer.

Node Version Manager (NVM)
/howto/install-nodejs-nvm-mac-m1/complete-installation-guide
71%
tool
Similar content

Node.js ESM Migration: Upgrade CommonJS to ES Modules Safely

How to migrate from CommonJS to ESM without your production apps shitting the bed

Node.js
/tool/node.js/modern-javascript-migration
71%
integration
Similar content

IB API Node.js: Build Trading Bots, TWS vs Client Portal Guide

TWS Socket API vs REST API - Which One Won't Break at 3AM

Interactive Brokers API
/integration/interactive-brokers-nodejs/overview
71%

Recommendations combine user behavior, content similarity, research intelligence, and SEO optimization