mongoexport Performance Optimization - Stop Waiting Hours for Exports

mongoexport performance sucks for several specific technical reasons that MongoDB doesn't make obvious. Based on real Stack Overflow threads and production experience, here's what actually kills performance and why your exports crawl at 500 docs per second on collections that should export way faster.

The Real Performance Killers

Single-Threaded Architecture: mongoexport is completely single-threaded. Even on a 16-core server, it'll max out one CPU core while the other 15 sit idle. This Stack Overflow thread shows someone waiting 12 hours to export 5.5% of a 130 million document collection. The MongoDB tools architecture never implemented parallel processing.

Terrible Memory Management: mongoexport is a memory-guzzling nightmare. With WiredTiger compression, it decompresses every fucking document into memory, does its thing, then throws it all away instead of streaming. I've watched it balloon to 14GB of RAM trying to export a collection that's 2GB compressed on disk. It's like watching someone fill up a swimming pool to wash their hands. Understanding WiredTiger storage explains why this is so inefficient.

Collection Scan Performance: Even with no query filters, mongoexport doesn't do efficient sequential reads. It performs scattered reads through WiredTiger's B-tree structure, which kills disk I/O performance. Someone with NVMe SSDs capable of 1GB/sec throughput was only getting 50MB/sec with mongoexport. The collection scanning behavior is fundamentally inefficient.

No Resume Capability: When mongoexport crashes (and it will), you start over from zero. No checkpointing, no resume functionality. Crash at 90% through your 48-hour export? You get to stare at this:

mongoexport --collection=massive_collection --out=data.json
2025-09-01T23:47:12.123+0000    connected to: mongodb://localhost/
2025-09-01T23:47:12.145+0000    exported 45372891 records
Killed

Then start over from 0 and contemplate your life choices.

Memory Usage Reality Check

Collection compression makes this worse. If your collection uses zlib compression (which is common), every document has to be decompressed during export. This happens in the same thread that's doing everything else, creating a CPU bottleneck even when your disk and network are underutilized. The compression algorithms all require CPU-intensive decompression.

Actual Numbers: A production export of a 15 million document collection (250GB on disk, compressed) required 8GB of RAM and took 18 hours. That's roughly 230 documents per second on hardware that should handle 10x that throughput.

The underlying getMore commands show the problem clearly:

command: getMore { getMore: 14338659261, collection: \"places\" } 
docsExamined:5369 numYields:1337 nreturned:5369 reslen:16773797 
protocol:op_query 22796ms

22.8 seconds to return 5,369 documents. That's 235 docs per second, and this was the optimized case.

Why Skip and Limit Don't Save You

The traditional workaround of using `--skip` and `--limit` to chunk exports doesn't work like you'd expect. MongoDB has to examine every document up to your skip value, so skip=10000000 means scanning 10 million documents just to start. This is a fundamental pagination limitation in MongoDB.

Skip Performance Reality:

• Skip 0: starts immediately
• Skip 1M: takes 5 minutes to start
• Skip 10M: takes 45 minutes to start
• Skip 50M: might never start

This makes parallel exports with skip/limit basically useless for large collections. Each process sits there scanning millions of documents it's going to ignore.

Since mongoexport is single-threaded garbage, the only real solution is running multiple processes in parallel. This isn't some theoretical optimization - it's been tested and works. Stack Overflow testing shows 6x speed improvements with 8 parallel processes. The technique is similar to MongoDB parallel bulk operations.

Query-Based Parallel Processing (Actually Works)

Instead of skip/limit, divide your collection by query ranges. This requires a field you can split on - ideally something with decent distribution. Understanding ObjectID structure helps with range splitting.

ObjectID-Based Splitting (Best Option):

## Calculate ObjectID ranges for time periods
## ObjectIDs from 2025-01-01 start with: 6585...
## ObjectIDs from 2025-06-01 start with: 6656...

mongoexport --query='{\"_id\":{\"$gte\":{\"$oid\":\"658500000000000000000000\"},\"$lt\":{\"$oid\":\"659000000000000000000000\"}}}' \
  --collection=orders --db=prod --out=orders_q1.json &

mongoexport --query='{\"_id\":{\"$gte\":{\"$oid\":\"659000000000000000000000\"},\"$lt\":{\"$oid\":\"65a000000000000000000000\"}}}' \
  --collection=orders --db=prod --out=orders_q2.json &

## Run 4-8 of these in parallel

Date-Based Splitting (If You Have Date Fields):

mongoexport --query='{\"created_at\":{\"$gte\":{\"$date\":\"2025-01-01\"},\"$lt\":{\"$date\":\"2025-03-01\"}}}' \
  --collection=events --db=analytics --out=events_q1.json &

mongoexport --query='{\"created_at\":{\"$gte\":{\"$date\":\"2025-03-01\"},\"$lt\":{\"$date\":\"2025-06-01\"}}}' \
  --collection=events --db=analytics --out=events_q2.json &

Hash-Based Splitting (For Even Distribution):

## Split by modulo on a numeric field
mongoexport --query='{\"user_id\":{\"$mod\":[4,0]}}' --collection=users --db=app --out=users_0.json &
mongoexport --query='{\"user_id\":{\"$mod\":[4,1]}}' --collection=users --db=app --out=users_1.json &
mongoexport --query='{\"user_id\":{\"$mod\":[4,2]}}' --collection=users --db=app --out=users_2.json &
mongoexport --query='{\"user_id\":{\"$mod\":[4,3]}}' --collection=users --db=app --out=users_3.json &

The `$mod` operator provides even distribution across processes. This approach works best with indexed fields for optimal query performance.

Performance Testing Results

Real-world testing on an 8-core server with a 200K document collection:

• 1 process: 32.7 seconds
• 2 processes: 16.5 seconds (2x speedup)
• 4 processes: 8.4 seconds (4x speedup)
• 8 processes: 5.1 seconds (6.4x speedup)

Beyond 8 processes, you hit diminishing returns as disk I/O becomes the bottleneck. The sweet spot is usually cores × 0.75 processes. Understanding CPU vs I/O bottlenecks helps optimize parallel configuration. Monitor with iostat to identify bottlenecks.

Python-Based Parallel Solution

For more control, use Python with multiprocessing. This approach uses PyMongo's parallel_scan (MongoDB 3.6+ with MMAPv1) or custom query splitting. The PyMongo documentation covers parallel processing patterns.

from multiprocessing import Pool
import subprocess
import json

def export_chunk(query_params):
    collection, db, query, output_file = query_params
    
    cmd = [
        'mongoexport', 
        '--collection', collection,
        '--db', db,
        '--query', json.dumps(query),
        '--out', output_file
    ]
    
    subprocess.run(cmd, check=True)
    return f\"Exported {output_file}\"

## Define your chunks
chunks = [
    ('orders', 'prod', {'_id': {'$gte': ObjectId('658500000000000000000000'), '$lt': ObjectId('659000000000000000000000')}}, 'orders_1.json'),
    ('orders', 'prod', {'_id': {'$gte': ObjectId('659000000000000000000000'), '$lt': ObjectId('65a000000000000000000000')}}, 'orders_2.json'),
    # Add more chunks...
]

## Run in parallel
with Pool(processes=8) as pool:
    results = pool.map(export_chunk, chunks)

Memory Optimization Per Process

Each mongoexport process still has the same memory problems, but now you're spreading the load. Monitor memory usage:

## Watch memory usage while parallel export runs
watch -n 1 'ps aux | grep mongoexport | grep -v grep'

If processes start getting OOMKilled, reduce parallelism or add swap. Each process can use 2-4GB of RAM depending on document size and complexity.

Combining Output Files

After parallel export, combine the files:

## JSON files (create array)
echo '[' > combined.json
find . -name \"chunk_*.json\" -exec cat {} \; | sed 's/$/,/' | sed '$ s/,$//' >> combined.json  
echo ']' >> combined.json

## CSV files (preserve header)
head -1 chunk_0.csv > combined.csv
tail -n +2 -q chunk_*.csv >> combined.csv

This parallel approach is the only proven way to make mongoexport perform acceptably on large collections. It's not elegant, but it works when you need to export millions of documents without waiting days.