Docker "No Space Left on Device" - Fix It Fast

Docker's "no space left on device" error is misleading bullshit. You check df -h and see tons of free space, but Docker still won't work. That's because Docker hides its data in places that df doesn't show you properly, and Docker never cleans up its own mess.

Where Docker Actually Hides Your Missing Disk Space

/var/lib/docker/ is where all your space went. Docker dumps everything here: container layers, images, volumes, build cache. I've seen fresh Docker installs grow from nothing to 30GB+ in a couple months of normal dev work. Check what's actually using space:

Check what's actually using space:

sudo du -sh /var/lib/docker/*
sudo du -sh /var/lib/docker/overlay2/* | sort -hr | head -10

What's eating your space in /var/lib/docker/:

• overlay2/ - Container filesystems (usually the biggest chunk)
• containers/ - Container logs and metadata (can get massive)
• image/ - Image data
• volumes/ - Your app data
• buildkit/ - Build cache (Docker never cleans this)

Container logs are probably your biggest problem. Docker logs everything to /var/lib/docker/containers/ and never rotates the files unless you tell it to. I've seen single containers dump 40GB+ of logs over a weekend.

Find log-heavy containers:

sudo find /var/lib/docker/containers/ -name "*.log" -exec du -Sh {} + | sort -rh | head -5

Docker image deduplication is a lie. Docker claims it shares layers between images, but half the time those "shared" layers end up orphaned and eating space anyway. Interrupted downloads and corrupted metadata leave behind GB of crap that nothing uses.

BuildKit cache is another space hog. Every docker build saves intermediate layers in /var/lib/docker/buildkit/ and Docker never expires them. I've seen build caches hit 20GB+ on active dev machines.

Hidden Space Problems That Don't Show in df

Inode Exhaustion happens when you run out of file system inodes (file entries) rather than disk space. Docker creates thousands of tiny files for container layers, and older filesystems can exhaust inodes while having plenty of space left. This is particularly common with ext4 filesystems on older systems.

Check for inode problems:

df -i
## Look for high "IUse%" numbers

The Linux filesystem documentation explains inode allocation and management across different filesystem types.

This is common on systems using ext4 with default inode ratios, especially with overlay2 storage driver creating many small files.

Filesystem-Specific Issues

Filesystem-Specific Issues vary by storage backend and Docker storage drivers:

• ext4: Can hit inode limits with Docker's small files
• xfs: Generally more resilient but can fragment with heavy layer usage
• APFS (macOS): Docker Desktop creates a disk image that doesn't shrink automatically
• NTFS (Windows): Layer deduplication fails, consuming excessive space

Docker Desktop Virtual Disk

Docker Desktop Virtual Disk (macOS/Windows) allocates space differently than Linux. The Docker.raw or ext4.vhdx files grow to accommodate containers but rarely shrink when containers are deleted. You might have 10GB of containers but a 60GB virtual disk file. The Docker Desktop troubleshooting guide covers virtual disk management and space reclamation.

When "Free Space" Isn't Actually Free

Modern filesystems reserve 5% of disk space for root, so when df shows you have space, you might not have enough for Docker operations. Docker daemon runs as root but creates files as various users, complicating space calculations.

Temporary Space During Operations

Temporary Space During Operations - Docker needs additional free space for:

• Layer extraction during image pulls (2x image size temporarily)
• Build operations that create intermediate layers
• Container startup when copying files
• Log rotation when it actually works

Network Filesystem Issues

Network Filesystem Issues (NFS, shared storage) where /var/lib/docker is on network storage can cause space reporting discrepancies. The local system might report free space, but the network filesystem is full.

LVM and Volume Management

LVM and Volume Management systems can show free space in volume groups that isn't allocated to the specific filesystem Docker uses. Your /var/lib/docker might be on a full logical volume while the system shows overall free space. The LVM documentation explains volume group and logical volume space management in detail.

The Container Logging Disaster

By default, Docker uses the `json-file` logging driver with no size limits. A single container that logs verbosely can fill your entire disk over a weekend.

Real-world disasters I've personally dealt with:

For more details on Docker logging problems, see the container logging best practices guide.

Weekend from hell: Production API with debug logging dumped 50GB of request/response logs over a weekend. Nobody even reads those JSON dumps but they killed our entire stack.

The infinite retry nightmare: Container couldn't connect to DB, so it logged "connection failed" every 100ms. Monday morning: 60GB of error messages in like 8 different languages because someone enabled i18n on error logging. Brilliant.

Docker Desktop space black hole: Colleague's MacBook had a 15GB Docker.raw file containing maybe 2GB of actual containers. Docker Desktop allocated space for deleted shit and never gave it back to macOS. Only fix was nuking everything.

How fast logs grow:

How fast logs grow:

• Dev containers with debug on: couple GB per day
• API services logging everything: 5-10GB daily
• Batch jobs that log every record: tens of GB per run
• Crash-looping containers: same error message over and over until disk dies

Log Location Investigation:

Log Location Investigation:

sudo du -sh /var/lib/docker/containers/*/ 
sudo du -sh /var/lib/docker/containers/*/*.log | sort -hr

This is probably the #1 cause of unexpected Docker space usage that catches teams off guard.

Docker Desktop vs Docker Engine Space Differences

Docker Desktop (Windows/Mac) uses a virtual machine approach where all Docker data lives inside a disk image file. This file grows dynamically but doesn't shrink automatically when you delete containers. The Desktop app shows container disk usage but doesn't account for image layer overhead or the virtual disk's allocated-but-unused space.

Docker Engine (Linux) stores everything directly in /var/lib/docker/ on the host filesystem. Space usage is more transparent but Docker's metadata tracking can become inconsistent, leading to "phantom" space usage where deleted containers still consume disk.

Anyway, that's where Docker hides your disk space. Now you can stop randomly running cleanup commands hoping something works.

For comprehensive space management strategies, refer to the Docker storage overview and troubleshooting documentation.

I've debugged this shit dozens of times. Here's what actually works, in order of how often they save your ass.

Figure Out What's Eating Your Space First

Don't just start deleting Docker images randomly. Find out where the space actually went:

## Check total Docker space usage
docker system df

## See detailed breakdown by type
docker system df -v

## Check actual disk usage in Docker directory
sudo du -sh /var/lib/docker/{overlay2,containers,buildkit,volumes}/*

docker system df shows you Docker's version of the truth, but it lies sometimes. du shows actual filesystem usage, which is often higher because Docker's accounting is fucked.

Red flags that mean you're fucked:

• Build cache over 5GB
• Container logs in the GB range
• Hundreds of <none> images from failed builds
• Stopped containers still taking up space
• Random volumes nobody remembers making

Solution 1: Fix the Logging Disaster First (Works 90% of the Time)

Container logs are probably your biggest space waster. This is what saves your ass most often:

## Find the log files that are eating your disk
sudo find /var/lib/docker/containers/ -name "*.log" -exec du -Sh {} + | sort -rh | head -5

## Truncate the massive ones immediately 
sudo find /var/lib/docker/containers/ -name "*.log" -exec truncate -s 0 {} \;

## Check how much space you just freed
df -h

Then prevent it from happening again:

## Configure log rotation in /etc/docker/daemon.json
sudo mkdir -p /etc/docker
echo '{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}' | sudo tee /etc/docker/daemon.json

## Restart Docker to apply
sudo systemctl restart docker

This fixes 90% of Docker space issues. Everything else is just cleanup.

Solution 2: Nuclear Option When Desperate

When log cleanup isn't enough or you're desperate:

## Stop all running containers first
docker stop $(docker ps -q)

## Nuclear cleanup - removes everything
docker system prune -a --volumes

## Check what you reclaimed
docker system df

Warning: This nukes everything. All containers, images, volumes - gone. But sometimes you just need to start fresh.

If those didn't work, it's probably the build cache:

## Check if BuildKit cache is the culprit
docker system df

## Nuclear build cache cleanup (often frees 5-20GB)
docker builder prune -a

## Clean up dangling images from failed builds
docker image prune

That covers the top 3 causes of Docker space issues. Everything else is just housekeeping.

Emergency Fixes When Docker Won't Start

If you're completely fucked and Docker won't even start:

## Stop Docker first
sudo systemctl stop docker

## Delete the log files manually (fastest space recovery)
sudo find /var/lib/docker/containers/ -name "*.log" -delete

## Clear system logs too if desperate  
sudo journalctl --vacuum-time=1d

## Restart Docker
sudo systemctl start docker

When You Need More Space Permanently

Move Docker's data to a bigger partition:

## Stop Docker first
sudo systemctl stop docker

## Move the entire directory
sudo mv /var/lib/docker /home/docker-data

## Create symlink or edit daemon.json with:
## "data-root": "/home/docker-data"

## Restart Docker
sudo systemctl start docker

Most Docker space problems boil down to: fix the logs, nuke everything, or clean the build cache.

Time to fix:

• Log cleanup: couple minutes, works most of the time
• Nuclear option: 5 minutes, works every time
• Build cache: 1 minute, usually frees 5-20GB
• Moving data directory: half hour if you have bigger storage

Set up log rotation and you'll never deal with this shit again.

Fixing Docker's space problems once doesn't prevent them from returning.

Here's how to set up your system so you don't waste time on this bullshit again.

Permanent Log Management Configuration

Global Docker daemon log configuration prevents the #1 cause of space issues

• runaway container logs:

Create or edit /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  },
  "storage-driver": "overlay2",
  "live-restore": true
}

Key configuration options:

• max-size:

Maximum size per log file (10m = 10 megabytes)

• max-file: Number of log files to keep (3 files × 10MB = 30MB max per container)
• compress:

Compresses rotated log files to save additional space

• live-restore: Keeps containers running during daemon restarts

After changing daemon configuration, restart Docker:

sudo systemctl restart docker

For more details on daemon configuration options, see the Docker daemon configuration reference.

Per-container log limits for specific applications:

## High-volume applications need tighter limits
docker run --log-opt max-size=5m --log-opt max-file=2 my-chatty-app

## Low-volume applications can have larger limits
docker run --log-opt max-size=50m --log-opt max-file=5 my-quiet-app

Alternative logging drivers that don't consume local disk.

See the logging drivers documentation for complete options:

## Send logs to syslog instead of files
docker run --log-driver=syslog --log-opt syslog-address=udp://1.2.3.4:514 app

## Use journald for systemd integration
docker run --log-driver=journald app

## Disable logging entirely (careful!)
docker run --log-driver=none app

Automated Maintenance Scripts

Daily cleanup script that runs during off-peak hours:

Create /usr/local/bin/docker-maintenance.sh:

#!/bin/bash

## Docker daily maintenance script
LOGFILE="/var/log/docker-maintenance.log"
DATE=$(date '+%Y-%m-%d %H:%M:%S')

echo "[$DATE] Starting Docker maintenance" >> $LOGFILE

## Remove stopped containers older than 24 hours
docker container prune -f --filter "until=24h"

## Remove unused networks
docker network prune -f

## Remove dangling images
docker image prune -f

## Clean build cache older than 48 hours
docker builder prune -f --filter until=48h

## Show disk usage after cleanup
echo "[$DATE] Docker space after cleanup:" >> $LOGFILE
docker system df >> $LOGFILE

echo "[$DATE] Maintenance completed" >> $LOGFILE

Set up cron job:

sudo chmod +x /usr/local/bin/docker-maintenance.sh

## Add to root's crontab (runs at 2 AM daily)
sudo crontab -e
## Add this line:
0 2 * * * /usr/local/bin/docker-maintenance.sh

For cron syntax reference, see the crontab documentation.

Weekly aggressive cleanup for development machines:

#!/bin/bash
## Weekly nuclear cleanup for dev environments
## /usr/local/bin/docker-weekly-cleanup.sh

## Stop all running containers
docker stop $(docker ps -q) 2>/dev/null

## Remove everything except volumes with important data
docker system prune -a -f --filter until=168h

## Clean up any remaining build cache
docker builder prune -a -f

echo "Weekly cleanup completed: $(docker system df)"

Monitoring and Alerting Setup

Disk space monitoring script that alerts before problems occur:

Create /usr/local/bin/docker-space-monitor.sh:

#!/bin/bash

## Docker space monitoring with email alerts
DOCKER_DIR="/var/lib/docker"
THRESHOLD=80  # Alert when over 80% full
EMAIL="admin@company.com"

## Check Docker directory disk usage
USAGE=$(df "$DOCKER_DIR" | awk 'NR==2 {print $5}' | sed 's/%//')

if [ "$USAGE" -gt "$THRESHOLD" ]; then
    # Log the issue
    logger "Docker storage $USAGE% full 
- threshold exceeded"
    
    # Send email alert (requires mailutils)
    echo "Docker storage on $(hostname) is $USAGE% full.

 Manual intervention required." | \
        mail -s "Docker Storage Alert 
- $(hostname)" "$EMAIL"
    
    # Optional: Auto-cleanup if over 90%
    if [ "$USAGE" -gt 90 ]; then
        docker system prune -f --filter until=24h
        logger "Auto-cleanup executed due to $USAGE% usage"
    fi
fi

Integrate with system monitoring (Prometheus, Grafana, Zabbix):

## Simple metric collection script
echo "docker_disk_usage_percent $(df /var/lib/docker | awk 'NR==2 {print $5}' | sed 's/%//')" \
    > /var/lib/prometheus/node-exporter/docker-disk.prom

Development Workflow Optimization

Use multi-stage builds to reduce final image sizes.

The multi-stage build documentation explains best practices:

## Multi-stage build reduces final image size
FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

FROM node:18-alpine AS runtime
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY src/ ./src/
CMD ["npm", "start"]

Optimize Docker build context with comprehensive .dockerignore.

See the .dockerignore reference:

node_modules/
npm-debug.log
.git
.gitignore
README.md
.env
.nyc_output
coverage
.nyc_output
.cache
dist/
build/
*.log

BuildKit configuration for better cache management:

## Enable BuildKit globally
export DOCKER_BUILDKIT=1

## Or configure in daemon.json
{
  "features": {
    "buildkit": true
  }
}

Development container best practices:

## Use smaller base images
FROM alpine:
3.18 instead of FROM ubuntu:
22.04

## Clean up in the same layer
RUN apt-get update && apt-get install -y package && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

## Don't install development tools in production images
FROM python:
3.11-slim-bullseye  # Instead of python:
3.11

Storage Infrastructure Improvements

Separate Docker data directory to dedicated partition:

## Create dedicated partition for Docker (during system setup)
## /dev/sdb1 mounted as /var/lib/docker

## Or use LVM for flexible space management
sudo lvcreate -L 50G -n docker-data vg0
sudo mkfs.ext4 /dev/vg0/docker-data
sudo mount /dev/vg0/docker-data /var/lib/docker

Configure different storage drivers based on use case:

• overlay2:

Default, good for most workloads

• devicemapper: Legacy, avoid if possible
• btrfs:

Good for systems already using btrfs

• zfs: Advanced features but more complex

Docker Desktop resource limits (macOS/Windows):

## Set reasonable limits in Docker Desktop preferences
## Memory: 8GB (instead of unlimited)
## Disk: 100GB (instead of growing forever)
## CPU: 4 cores (don't hog the entire machine)

Team and CI/CD Best Practices

Establish team conventions:

• Always use .dockerignore files
• Set log limits on all containers
• Regular image cleanup in shared development environments
• Use specific image tags, not :latest

CI/CD pipeline optimization:

## GitHub Actions example with cleanup
- name:

 Clean up Docker before build
  run: docker system prune -f
  
- name:

 Build with BuildKit
  run: |
    export DOCKER_BUILDKIT=1
    docker build --no-cache -t app:${{ github.sha }} .
    
- name:

 Clean up after build
  run: docker system prune -f
  if: always()

Image registry management:

• Use registry cleanup policies
• Implement image tagging strategies that enable automatic cleanup
• Don't keep every development build forever

Documentation and runbooks:

• Document disk space requirements for different environments
• Create runbooks for space emergencies
• Train team on proper Docker resource management

Advanced Prevention Techniques

Container resource limits prevent runaway processes:

## Memory limits prevent containers from consuming all RAM
docker run -m 512m --memory-swap 1g my-app

## CPU limits prevent CPU starvation
docker run --cpus="1.5" my-app

## Disk I/O limits
docker run --device-read-bps /dev/sda:1mb --device-write-bps /dev/sda:1mb my-app

Volume management strategies:

## Named volumes are easier to manage than anonymous ones
docker run -v myapp-data:/data my-app

## Regular volume cleanup
docker volume ls -qf dangling=true | xargs docker volume rm

Network cleanup automation:

## Networks can accumulate and consume metadata space
docker network prune -f

The key to preventing Docker space issues is proactive configuration and monitoring.

Set up proper logging, implement regular cleanup, and monitor usage trends before problems occur.

Most Effective Prevention Stack:

1. Global log rotation (daemon.json configuration)

• 90% of space issues

2. Daily automated cleanup (cron job)

• Prevents accumulation

3. Disk space monitoring (alerting before full)

• Early warning

4. Team education (proper Docker practices)

• Reduces problem frequency

Setting this up takes about 2 hours (or 6 hours if you hit weird permission issues) but saves days of troubleshooting over the lifetime of your Docker infrastructure.