Docker "No Space Left on Device" Error: AI-Optimized Technical Reference

Critical Context

Severity Indicators

• Critical: Production deadlines killed by deployment failures
• High Impact: 50GB+ hidden in /var/lib/docker despite showing "plenty" of free space
• Emergency: Docker won't start, preventing all container operations
• Common Scale: Fresh Docker installs grow to 30GB+ within months of normal development

Frequency and Impact

• 90% of cases: Container logs are the primary space consumer
• Real-world disasters: 50GB weekend log dumps, 60GB crash-loop error messages
• Time to fix: Log cleanup (2 minutes), nuclear cleanup (5 minutes), build cache cleanup (1 minute)

Root Cause Analysis

Primary Space Consumers (by frequency)

1. Container logs (/var/lib/docker/containers/) - 90% of space issues

   • No default size limits or rotation
   • Single containers can generate 40GB+ logs over weekends
   • Debug logging can produce GB per day
   • API services with verbose logging: 5-10GB daily

2. Build cache (/var/lib/docker/buildkit/)

   • Never expires by default
   • Commonly reaches 20GB+ on active development machines
   • Stores intermediate layers from every docker build

3. Container layers (/var/lib/docker/overlay2/)

   • Orphaned layers from interrupted downloads
   • Failed "deduplication" leaving unused shared layers
   • Corrupted metadata creating phantom space usage

Hidden Space Problems

• Inode exhaustion: Docker creates thousands of tiny files, exhausting file system inodes while space remains
• Reserved space: Filesystems reserve 5% for root, reducing available space
• Virtual disk files: Docker Desktop creates disk images that grow but never shrink automatically
• Network filesystem discrepancies: NFS/shared storage showing local free space while network storage is full

Configuration Requirements

Critical Logging Configuration

File: /etc/docker/daemon.json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  },
  "storage-driver": "overlay2",
  "live-restore": true
}

Space calculation: 3 files × 10MB = 30MB maximum per container
Implementation requirement: Restart Docker daemon after configuration

Production Settings That Actually Work

• High-volume applications: max-size=5m, max-file=2
• Low-volume applications: max-size=50m, max-file=5
• Alternative logging drivers: syslog, journald, or none (to eliminate local disk usage)

Diagnostic Commands

Space Investigation (Run in sequence)

# Docker's version of truth (often lies)
docker system df -v

# Actual filesystem usage (more reliable)
sudo du -sh /var/lib/docker/{overlay2,containers,buildkit,volumes}/*

# Find massive log files
sudo find /var/lib/docker/containers/ -name "*.log" -exec du -Sh {} + | sort -rh | head -5

# Check for inode exhaustion
df -i

Red Flags Indicating Critical Problems

• Build cache over 5GB
• Container logs in GB range
• Hundreds of <none> images from failed builds
• Stopped containers still consuming space
• Random volumes with unknown ownership

Solution Procedures

Emergency Recovery (When Docker Won't Start)

# Stop Docker daemon
sudo systemctl stop docker

# Delete log files manually (fastest space recovery)
sudo find /var/lib/docker/containers/ -name "*.log" -delete

# Clear system logs if desperate
sudo journalctl --vacuum-time=1d

# Restart Docker
sudo systemctl start docker

Standard Cleanup (Works 90% of cases)

# Truncate massive log files immediately
sudo find /var/lib/docker/containers/ -name "*.log" -exec truncate -s 0 {} \;

# Nuclear cleanup when desperate (removes everything)
docker stop $(docker ps -q)
docker system prune -a --volumes

# Build cache cleanup (often frees 5-20GB)
docker builder prune -a

Permanent Solution (Moving Data Directory)

# Stop Docker
sudo systemctl stop docker

# Move entire directory
sudo mv /var/lib/docker /home/docker-data

# Create symlink or edit daemon.json with:
# "data-root": "/home/docker-data"

# Restart Docker
sudo systemctl start docker

Prevention Automation

Daily Maintenance Script

Location: /usr/local/bin/docker-maintenance.sh
Schedule: 2 AM daily via cron
Actions:

• Remove stopped containers older than 24 hours
• Remove unused networks
• Remove dangling images
• Clean build cache older than 48 hours

Monitoring Thresholds

• Warning: 80% disk usage
• Critical: 90% disk usage (triggers auto-cleanup)
• Emergency: Above 95% (manual intervention required)

Resource Requirements

Time Investment

• Initial setup: 2 hours (6 hours with permission issues)
• Emergency fixes: 2-5 minutes
• Permanent configuration: 30 minutes
• ROI: Saves days of troubleshooting over infrastructure lifetime

Expertise Requirements

• Basic: Understanding of filesystem permissions and Docker daemon configuration
• Intermediate: Cron job configuration and log rotation setup
• Advanced: LVM/storage management for dedicated partitions

Space Allocation Guidelines

• Development: Plan for 3-5x expected image sizes plus logs
• Typical web app: 20-50GB allocation
• Data processing workloads: Hundreds of GB
• Buffer: Add 50% to baseline measurements

Critical Warnings

Breaking Points

• UI breaks at 1000 spans: Makes debugging large distributed transactions impossible
• Container crash during space exhaustion: Applications rarely recover gracefully
• Build failures with marginal space: Docker needs 2x final image size temporarily
• Docker Desktop space black holes: Virtual disks grow but never shrink automatically

Common Misconceptions

• "df shows free space": Docker's space calculation differs from standard filesystem tools
• "Shared layers save space": Deduplication often fails, leaving orphaned layers
• "Cleanup commands always work": Docker's accounting can be inconsistent, requiring daemon restarts

Production Gotchas

• Default log driver has no limits: Will consume entire disk without intervention
• BuildKit cache never expires: Accumulates indefinitely without manual cleanup
• Image deletion doesn't free space: Shared layers may still be referenced
• Network filesystems show incorrect free space: Local tools report local filesystem, not network storage

Decision Support Matrix

Scenario	Time Cost	Difficulty	Success Rate	Risk Level
Log cleanup	2 minutes	Easy	90%	Low
Nuclear cleanup	5 minutes	Easy	100%	High (data loss)
Build cache cleanup	1 minute	Easy	95%	Low
Moving data directory	30 minutes	Medium	95%	Medium
Full prevention setup	2-6 hours	Medium	99%	Low

Implementation Checklist

Immediate Actions (Emergency)

• Check actual space usage with du commands
• Identify log files consuming excessive space
• Truncate or delete massive log files
• Verify Docker can start after cleanup

Short-term Configuration (Same day)

• Configure log rotation in daemon.json
• Restart Docker daemon
• Test log rotation with verbose container
• Set up basic monitoring

Long-term Prevention (Within week)

• Implement daily cleanup cron job
• Configure disk space monitoring with alerts
• Train team on proper Docker practices
• Document emergency procedures

Optimization (Ongoing)

• Review and optimize Dockerfiles for smaller images
• Implement multi-stage builds
• Configure CI/CD cleanup procedures
• Monitor space usage trends for capacity planning

Alternative Solutions and Trade-offs

Logging Driver Alternatives

• syslog: Eliminates local disk usage but requires syslog infrastructure
• journald: Integrates with systemd but still consumes local space
• none: Eliminates all logging (dangerous for debugging)

Storage Driver Comparison

• overlay2: Default, most space-efficient, best performance
• devicemapper: Legacy, can waste space, avoid if possible
• btrfs/zfs: Advanced features but increased complexity

Platform-Specific Considerations

• Docker Desktop: Virtual disk management, different space reporting
• Linux: Direct filesystem access, more predictable behavior
• Windows/WSL2: Additional virtualization layer complications