MongoDB Replica Sets - Stop Your Database From Crashing at 3AM

Look, I've been woken up at 3AM too many times by MongoDB replica sets shitting the bed. Let me save you some pain and show you how to set these up so they actually work when your traffic spikes.

What Replica Sets Actually Are (Skip If You Know)

Three MongoDB servers pretending to be one. One handles writes (primary), the other two copy everything and can handle reads (secondaries). When the primary dies, the other two vote on who becomes the new primary. Takes about 10 seconds in MongoDB 8.0, used to take 15+ in older versions.

The election process is where shit usually goes wrong. Network hiccups, resource exhaustion, or bad config can cause election storms where your cluster spends more time electing leaders than serving requests.

The Config That Actually Works in Production

Three nodes, different data centers. Period.

Don't get cute with single-AZ deployments unless you enjoy explaining to your boss why the database is down. We run:

• Primary: us-east-1a (m5.2xlarge, 32GB RAM, 1TB gp3)
• Secondary 1: us-east-1b (same specs)
• Secondary 2: us-west-2a (same specs, disaster recovery)

Total monthly cost on AWS: about $800 including storage. Atlas would be $1,200+ for the same thing, but you get backups and support included.

Don't use arbiters. Seriously. I see this recommended everywhere and it's garbage advice. Arbiters save you maybe $300/month but give you one less copy of your data. When your secondary dies and you're running on primary + arbiter, you're one failure away from total data loss.

Hardware That Won't Let You Down

Memory: Your working set MUST fit in RAM. If it doesn't, MongoDB becomes dog slow. We size for 2x our current working set because growth happens fast. Check the MongoDB memory requirements guide for detailed memory planning.

Storage: NVMe SSDs or you're wasting your time. We learned this the hard way when our replica set fell 30 seconds behind during a traffic spike because we cheaped out on spinning disks. See MongoDB storage recommendations for specific storage requirements.

Network: Gigabit minimum between nodes. 10Gb if you can afford it. Network latency over 10ms between replica set members will bite you during elections. Refer to MongoDB networking requirements for detailed network specifications.

CPU: 8+ cores. MongoDB is single-threaded for writes but uses multiple threads for everything else. Check MongoDB CPU recommendations for optimal CPU configurations.

MongoDB 8.0: Actually Worth Upgrading

MongoDB 8.0 fixed some annoying shit that was paging us regularly in 7.x:

• Elections finish in 8-12 seconds instead of 15-20 seconds
• Replication lag dropped from 2-3 seconds to under 1 second for our workload
• Memory usage is more predictable (fewer OOM kills)

MongoDB 8.0 delivers massive performance improvements across all major workloads - 36% faster reads, 32% faster mixed workloads, and 56% faster bulk writes. These aren't marketing numbers; they're from actual benchmark tests using industry-standard YCSB and real customer workloads.

The upgrade from 7.0 to 8.0 took us a weekend with rolling restarts. Zero downtime, just had to deal with some connection pool churn during the restarts. Follow the MongoDB upgrade procedures for detailed upgrade steps. Also check the compatibility changes before upgrading.

Read Preferences That Don't Suck

primaryPreferred: Use this for most stuff. Reads go to primary unless it's overloaded, then failover to secondaries.

secondary: Only use this for analytics queries or stuff where stale data is OK. Your secondaries are usually 500ms-2 seconds behind the primary.

nearest: Sounds smart but can cause weird behavior if your app logic assumes consistent reads. Stick with primaryPreferred unless you have a specific use case.

We route all user-facing reads to primaryPreferred and analytics/reporting to secondary. Works well, just monitor replication lag or your reports will be stale as hell. Read more about MongoDB read preferences and read preference mechanisms for detailed configuration options.

Security (Or How Not to Get Pwned)

Authentication: Turn it on. Seriously. I've seen production MongoDB clusters running wide open on the internet. Don't be that guy. Follow the MongoDB authentication guide for proper auth setup.

## Generate a keyfile for internal auth
openssl rand -base64 756 > /etc/mongodb-keyfile
chmod 400 /etc/mongodb-keyfile  
chown mongodb:mongodb /etc/mongodb-keyfile

TLS: Enable it between clients and MongoDB, and between replica set members if you're paranoid about network sniffing. See MongoDB TLS configuration for detailed TLS setup.

Networks: Put your MongoDB servers in a private subnet. Only allow access from your app servers. Use security groups/firewalls to lock it down. Check the network security recommendations for best practices.

Users: Create specific database users with minimal permissions. Don't give your app the admin password. Learn about MongoDB authorization and built-in roles for proper user management.

Monitoring That Actually Helps

What to monitor:

• Replication lag (alert if >2 seconds)
• Connection count (we max out around 500 concurrent)
• Memory usage (page faults = death)
• Elections (alert on any election - they should be rare)

Tools that work:

• MongoDB Compass for quick checks
• Prometheus + Grafana for historical trends
• PMM from Percona for detailed monitoring
• MongoDB Atlas built-in monitoring if you're using Atlas
• DataDog MongoDB integration for enterprise monitoring
• New Relic MongoDB quickstart for APM integration

PMM provides comprehensive MongoDB monitoring with real-time metrics, query analysis, and performance insights that actually help you debug production issues.

Don't monitor everything. We tried that and got alert fatigue. Focus on the metrics that indicate actual problems, not just "database is busy."

Key monitoring resources:

• MongoDB Performance Best Practices
• Replica Set Monitoring Tutorial
• MongoDB Metrics That Matter

Common Fuckups and How to Avoid Them

Oplog too small: We size ours to hold 24 hours of writes. Default is 5% of disk space which might not be enough.

Mixed hardware: Don't put your primary on a beast server and secondaries on tiny instances. When failover happens, your new primary will be slow as shit.

Single AZ: Network outages happen. Don't put all your eggs in one availability zone.

No monitoring: You'll find out your replica set is having problems when your app starts timing out. Set up monitoring before you go to production.

The goal is a replica set that just works. Boring is good when it comes to databases.

Your replica set will break. It's not "if", it's "when". I've been paged at 2AM more times than I can count because of MongoDB shitting the bed. Here's how to debug the common failures and get back to sleep.

The "Holy Shit, Replication Lag is 30 Seconds" Problem

This happened to us during Black Friday 2024. Suddenly our analytics dashboard was showing data from 30 seconds ago, users were complaining about stale product availability, and our on-call engineer was having a panic attack.

What causes replication lag:

• Network between nodes is saturated (most common)
• Secondary hardware can't keep up with primary writes
• Oplog is too small and rolling over
• Locks on secondaries from long-running queries

Check the MongoDB replication lag troubleshooting guide and oplog size recommendations for detailed diagnosis steps.

Debug replication lag like this:

// Check current lag
db.runCommand("replSetGetStatus").members.forEach(
  function(member) {
    if (member.state === 2) { // Secondary
      var lag = (new Date() - member.optimeDate) / 1000;
      print(member.name + " lag: " + lag + " seconds");
    }
  }
)

How we fixed it:

1. Scaled up secondary instances from m5.large to m5.xlarge (more CPU/memory)
2. Upgraded network from 1Gb to 10Gb between data centers
3. Increased oplog size from 5GB to 50GB
4. Moved analytics queries to a dedicated secondary with read tags

Additional resources that helped:

• MongoDB Replication Performance
• Troubleshooting Replica Sets
• Atlas Performance Tuning
• Network Requirements Guide

Total downtime: 0 minutes. Total time to fix: 6 hours. Total AWS bill increase: $400/month.

Understanding MongoDB replication flow is crucial for debugging lag issues - the primary handles all writes and replicates to secondaries asynchronously.

Election Hell: When Your Replica Set Can't Pick a Leader

We had a replica set that was holding elections every 30 seconds for 2 hours straight. No writes could complete, the app was throwing "NotPrimaryError" constantly, and I was losing my shit trying to figure out what was wrong.

What causes election storms:

• Network flakiness between nodes (usually the culprit)
• Primary getting OOM killed repeatedly
• Misconfigured replica set priorities
• Split-brain scenarios (rare but terrifying)

For comprehensive election troubleshooting, check MongoDB election diagnostics and election timeout tuning guide.

The "MongoNetworkTimeoutError" nightmare:

MongoNetworkTimeoutError: connection timed out
at connectionFailureError (/app/node_modules/mongodb/lib/core/connection/pool.js:596:6)
at Pool.<anonymous> (/app/node_modules/mongodb/lib/core/connection/pool.js:434:15)

This error means your app can't reach the primary, which probably means an election is happening.

How to debug elections:

## Check election history
db.adminCommand("replSetGetStatus").members.forEach(function(member) {
  print(member.name + " - uptime: " + member.uptime + " - state: " + member.stateStr)
})

## Look for high election counts in logs
grep "election succeeded" /var/log/mongodb/mongod.log

Our fix:

• Moved to dedicated 10Gb network links between replica set members
• Set electionTimeoutMillis to 20000 (20 seconds) from default 10 seconds
• Added more monitoring for network latency between nodes
• Implemented priority-based elections to prefer specific nodes

Election debugging resources:

• Election Process Documentation
• Replica Set Configuration Reference
• Network Diagnostics Guide

Elections dropped from every 30 seconds to maybe once per month.

Connection Pool Bullshit: "No Suitable Servers Found"

This error makes me want to throw my laptop out the window:

MongoServerSelectionError: No suitable servers found (`serverSelectionTryOnce` set): 
[connection refused calling ismaster on 'mongodb-primary:27017']

What this actually means:

• Your connection pool is full
• MongoDB is rejecting new connections
• Network between app and MongoDB is fucked
• All replica set members are unreachable

How we fixed connection pool exhaustion:

// Node.js connection config that actually works
const client = new MongoClient(uri, {
  maxPoolSize: 50,        // Default is 100, way too high
  minPoolSize: 5,         // Keep some connections warm
  maxIdleTimeMS: 30000,   // Close idle connections after 30 seconds  
  serverSelectionTimeoutMS: 5000, // Fail fast, don't wait 30 seconds
  socketTimeoutMS: 20000, // Individual operation timeout
});

Monitor your connection pools:

## Check current connections
db.serverStatus().connections
## active: 42, current: 45, available: 954

## If available is low, you're in trouble

Learn more about MongoDB connection pooling configuration and best practices.

Connection pooling manages database connections efficiently - when configured properly, it prevents connection exhaustion and reduces latency.

Performance Shit That Actually Matters

Memory: Get It Right or Die

MongoDB loves RAM. If your working set doesn't fit in memory, you're fucked. Here's how to not fuck it up:

## Check if you're swapping to disk (very bad)
db.serverStatus().wiredTiger.cache["bytes currently in the cache"] 
db.serverStatus().wiredTiger.cache["maximum bytes configured"]

## If you're hitting the max cache size, scale up now

WiredTiger cache usage directly impacts query performance - monitor cache hit ratios and eviction rates to prevent memory-related slowdowns.

We found out the hard way that our 16GB instances couldn't handle our working set. Queries went from 50ms to 2 seconds overnight when we hit the memory limit.

Solution: Upgraded to 64GB instances, set WiredTiger cache to 32GB (50% of RAM). Problem solved, $600/month more expensive. Learn more about WiredTiger cache tuning and memory management best practices.

Write Performance: It's All About the Primary

All writes go to the primary, so if the primary is slow, everything is slow.

Write concern settings that won't kill performance:

• w:1 - Fast but dangerous, data might be lost if primary crashes
• w:majority - Safe but slow, waits for majority of nodes to acknowledge
• w:0 - Fire and forget, don't use this in production

We use w:1 for user actions (likes, comments) and w:majority for financial transactions. Compromise between speed and safety. Read more about write concern and write concern performance implications.

Index hell:
Every additional index slows down writes. We had 47 indexes on one collection and writes were taking 500ms. Dropped unnecessary indexes, writes dropped to 50ms. Check the MongoDB indexing guide and index performance strategies for optimal index management.

Disaster Recovery: When Everything Goes to Hell

Backups Are Not Optional

Replica sets protect against hardware failure, not human stupidity. We accidentally deleted 50,000 user records with a bad query. All three replica set members deleted the data instantly.

What actually works:

• Atlas backups (point-in-time recovery, costs extra)
• EBS snapshots of your data volumes (cheap, restore takes hours)
• `mongodump` to S3 (slow but simple)

For comprehensive backup strategies, see MongoDB backup methods and backup best practices.

Test your backups. We found out our backup restore process was broken when we actually needed it. Don't be us.

Multiple backup strategies provide different recovery time objectives - Atlas offers point-in-time recovery, while self-hosted requires more manual setup.

Multi-Region: Expensive but Worth It

We run secondaries in us-east-1, us-west-2, and eu-west-1. When AWS us-east-1 shit the bed in 2024, our app kept running because the secondary in us-west-2 became primary automatically.

Geographic distribution provides disaster recovery but comes with added complexity - elections must consider network partitions and regional latencies.

Cost: Extra $800/month for cross-region data transfer and additional instances.
Value: App stayed up when our main region was down for 4 hours.

When to Stop Using Replica Sets

Signs you need to shard:

• Primary maxed out at 100% CPU constantly
• Working set larger than largest available instance memory
• Write throughput exceeding 50,000 ops/second
• Data size over 2TB (maintenance windows become painful)

Sharding is complex but necessary for serious scale. We made the jump when our dataset hit 5TB and queries started taking 30+ seconds. Learn about MongoDB sharding and sharding strategies before making the transition.