memory_profiler - Track Memory Usage Line-by-Line in Python

memory_profiler solves the problem of "where the fuck is my Python script eating all the RAM?" While cProfile tells you what's slow, memory_profiler tells you what's fat. Simple as that.

It's Dead But Still Works

Version 0.61.0 from November 2022 is the final release. The maintainers officially threw in the towel - they're not fixing bugs or answering issues. But here's the thing: it still works perfectly. 4.5k GitHub stars don't lie. The code is stable, battle-tested, and handles Python 3.5+ without breaking.

How It Actually Works

Two ways to use it: slap @profile decorators on functions for line-by-line analysis, or use mprof to spy on running processes externally. The decorator approach shows you exactly which line ate 500MB. The external monitoring catches memory leaks in production without touching your code.

It uses psutil under the hood for cross-platform memory readings. Unlike tracemalloc (which only tracks Python objects), memory_profiler shows real system memory including C extensions and whatever numpy is doing behind the scenes.

When You Actually Need This

memory_profiler is handy when your Python script eats more RAM than Chrome. Specifically:

• Your data processing pipeline randomly OOMs on production data
• Your web app slowly leaks memory until it dies at 3am (I once had a Flask app that leaked 50MB every hour - took three weeks to find the culprit: a pandas DataFrame that wasn't getting garbage collected because of a circular reference. memory_profiler pinpointed it in 20 minutes.)
• Your ML training crashes after 6 hours with "cannot allocate memory"
• You're debugging scientific code that should use 2GB but somehow needs 16GB

Works great with Jupyter notebooks via %mprun and %memit magic commands. Load it with %load_ext memory_profiler and you're off to the races.

Production Reality

Line-by-line profiling slows things down noticeably - expect 2-10x overhead depending on your code. I once had memory_profiler running on production and forgot about the overhead. Our API response time went from 200ms to 2 seconds. Customer called screaming about the slowdown while I frantically figured out why everything was broken. For production monitoring, use mprof to watch from outside. The measurements vary between runs because Python's garbage collector does whatever it wants, but the trends are reliable enough to catch real problems.

Getting Started

`pip install memory_profiler` and you're done. Pulls in psutil as a dependency, which handles the cross-platform memory measurements. If you're using conda, there's a conda-forge package too.

Line-by-Line Analysis (The Main Event)

Slap @profile on any function, then run python -m memory_profiler script.py. You'll get output showing line numbers, memory usage after each line, and the delta. Sometimes the output is confusing because Python's garbage collector runs between lines, but you'll spot the real memory hogs.

@profile
def load_huge_data():
    data = pandas.read_csv('dataset.csv')  # +500MB (ouch)
    filtered = data[data.value > 100]      # +200MB (copy strikes again)
    result = filtered.groupby('id').sum()  # +50MB (reasonable)
    return result

External Process Monitoring (Production Friendly)

Use mprof run python script.py to watch from outside without touching your code. Follow with mprof plot to get a matplotlib graph. This is your go-to for production debugging since it doesn't slow things down as much.

The graphs sometimes lie when multiprocessing is involved - RSS measurements double-count shared memory. Use --backend psutil_pss to fix this, assuming your system supports PSS (Linux does, others might not).

Memory Backends and Platform Gotchas

You've got five backends to choose from: 'psutil' (default), 'psutil_pss', 'psutil_uss', 'posix', and 'tracemalloc'. Stick with 'psutil' unless you're debugging in Docker (use 'psutil_pss') or you actually understand the difference between RSS and USS (most of us don't). RSS is wrong in Docker containers because shared memory gets counted multiple times. PSS fixes this by dividing shared pages among processes. USS only counts private memory.

Windows works but memory measurements are less reliable than Linux. macOS works fine but sometimes reports negative memory usage deltas, which makes zero sense but happens anyway.

Multiprocessing Hell

Use --include-children to aggregate child process memory or --multiprocess to track each separately. Neither works perfectly with Python's multiprocessing module because of how it forks processes. You'll get weird spikes that don't reflect reality.

Jupyter Integration

Works with Jupyter notebooks, which is nice when you're prototyping. Load with %load_ext memory_profiler, then use %mprun to profile functions or %memit to time memory usage like `%timeit` but for RAM.

%load_ext memory_profiler
%mprun -f my_function my_function(data)
%memit [x**2 for x in range(10000)]

Memory Breakpoints (Actually Useful)

Use --pdb-mmem=1000 to drop into the debugger when memory hits 1GB. Great for catching runaway allocations before they kill your server. Set it lower than your system's memory limit or you'll be debugging a dead process. On a typical data processing script (500MB dataset, pandas operations), memory_profiler added about 4x overhead - your 30-second script becomes 2 minutes. Plan accordingly.