Django Celery Redis Docker: Production-Ready Background Task Processing

Critical Architecture Overview

Stack Components: Django web servers + Celery workers + Redis message broker + PostgreSQL database + Docker containers

Performance Impact: Response time improvement from 850ms to 45ms for report generation, peak CPU usage reduced from 85% to 35%, eliminated user timeout errors (from ~50/day to 0)

Scaling Capacity: Handles 20k daily active users, supports concurrent user capacity roughly doubled through background task processing

Configuration Requirements

Redis Production Settings

redis:
  command: redis-server --appendonly yes --save 60 1000 --maxmemory 512mb --maxmemory-policy allkeys-lru
  volumes:
    - redis_data:/data
  restart: unless-stopped

Critical Parameters:

• --appendonly yes: Prevents message loss during crashes (learned after losing 2000 queued tasks)
• --maxmemory 512mb: Prevents Redis consuming all server memory (crashed production twice)
• --maxmemory-policy allkeys-lru: Evicts oldest data when memory full

Celery Worker Configuration

# Production settings that prevent memory leaks
CELERY_WORKER_MAX_TASKS_PER_CHILD = 1000  # Workers grow to 800MB+ without this
CELERY_WORKER_PREFETCH_MULTIPLIER = 1     # Prevents worker task hoarding
CELERY_TASK_ACKS_LATE = True              # Acknowledge after completion
CELERY_RESULT_EXPIRES = 3600              # Results expire after 1 hour
CELERY_TASK_IGNORE_RESULT = True          # Don't store results if not needed

Resource Limits (prevents OOM kills):

worker:
  deploy:
    resources:
      limits:
        memory: 1.5G  # Generous limit prevents random deaths
      reservations:
        memory: 512M

Database Connection Management

DATABASES = {
    'default': {
        'CONN_MAX_AGE': 300,  # Connection pooling
        'OPTIONS': {
            'MAX_CONNS': 4,   # Max 4 connections per worker
        }
    }
}

Connection Limits: PostgreSQL defaults to 100 connections. Formula: 8 web workers × 5 connections + 6 Celery workers × 5 connections + monitoring = limit exceeded. Must increase to 200+ connections.

Critical Failure Modes and Solutions

Docker Networking Failures

Problem: Workers crash with "Connection refused" - everything works locally but fails in containers
Root Cause: Using localhost:6379 instead of Docker service names
Fix: Use service names: redis://redis:6379/1 not redis://localhost:6379/1
Time Cost: 4 hours first occurrence, 30 minutes every subsequent time

Synchronous Task Execution (Silent Failure)

Problem: Tasks execute in web process instead of background workers
Root Cause: CELERY_TASK_ALWAYS_EAGER = True in settings
Detection: Queue slow task (time.sleep(10)), if web request blocks for 10 seconds, eager mode is active
Fix: Set CELERY_TASK_ALWAYS_EAGER = False or remove setting entirely

Redis Message Loss

Problem: Redis crashes eat all queued tasks, workers lose connection
Root Cause: Default Redis config doesn't persist to disk
Consequence: All background work disappears during restarts/crashes
Solution: Enable AOF persistence and volume mounting as shown in configuration

Worker Memory Growth (OOM Kills)

Problem: Workers start at 100MB, grow to 800MB+ over days, get killed by Docker
Root Cause: Python garbage collection imperfect, memory leaks accumulate
Detection: docker stats to monitor growth, docker exec worker ps aux for processes
Solutions:

• Force worker recycling: CELERY_WORKER_MAX_TASKS_PER_CHILD = 500
• Prevent task hoarding: CELERY_WORKER_PREFETCH_MULTIPLIER = 1
• Set generous Docker memory limits: 1.5G minimum

PostgreSQL Connection Exhaustion

Problem: "FATAL: too many connections for role" errors
Calculation: 8 web workers × 5 connections + 6 workers × 5 connections + monitoring = 70+ connections (PostgreSQL default: 100)
Solutions:

1. Increase PostgreSQL limit: ALTER SYSTEM SET max_connections = 200;
2. Limit per-process connections: MAX_CONNS: 4 in Django settings
3. Nuclear option: connections.close_all() in tasks

Task Import Failures

Problem: Tasks fail with ImportError: No module named 'myapp.models'
Root Cause: Different PYTHONPATH or DJANGO_SETTINGS_MODULE between containers
Solution: Ensure identical environment variables in web and worker containers
Debug: docker exec worker python -c "import myapp.models; print('works')"

Infinite Task Pending State

Problem: Tasks show as queued but never execute, no error messages
Cause: Task routing doesn't match worker queue configuration
Debug Commands:

docker exec worker celery -A myproject inspect ping
docker exec worker celery -A myproject inspect active_queues
docker exec worker celery -A myproject inspect active

Production Scaling Patterns

Queue-Based Auto Scaling

Trigger Points:

• Queue length > 500: Scale to 10 workers (heavy load)
• Queue length > 100: Scale to 5 workers (medium load)
• Queue length < 10: Scale to 2 workers (light load)

Specialized Worker Pools

# Fast tasks (email, notifications) - high concurrency
worker-fast:
  command: celery -A core worker -Q fast --concurrency=8

# CPU intensive (image processing) - low concurrency, high CPU
worker-cpu:
  command: celery -A core worker -Q cpu_intensive --concurrency=2
  resources:
    limits:
      cpus: '4.0'

# IO intensive (downloads, API calls) - very high concurrency
worker-io:
  command: celery -A core worker -Q io_intensive --concurrency=20

Monitoring Requirements

Critical Metrics:

• Queue depth: Scale workers when > 100 tasks for > 5 minutes
• Worker memory usage: Alert when > 1GB per worker
• Task failure rate: Alert when > 5% failure rate
• Redis memory usage: Alert when > 80% of limit
• Database connections: Alert when > 80% of max_connections

Health Check Commands:

# Worker health
celery -A core inspect ping

# Active tasks
celery -A core inspect active

# Redis memory
redis-cli info memory

# Database connections
SELECT count(*) FROM pg_stat_activity;

Resource Investment Reality

Time Costs:

• Initial setup: 3 months to get production-stable configuration
• Docker networking issues: 4 hours first time, 30 minutes recurring
• Memory leak debugging: 2-3 hours per incident
• Import/path issues: 1-2 hours typical resolution

Infrastructure Costs (monthly):

• Redis instance: ~$15 (t3.small)
• Additional monitoring: ~$10-25
• Increased database capacity: ~$12+ (connection limits)

Expertise Requirements:

• Docker networking knowledge (critical)
• Redis persistence and memory management
• PostgreSQL connection pooling
• Python memory profiling for leak detection

When NOT to Use This Stack

Skip for:

• Basic CRUD applications with < 1000 daily users
• Operations completing in < 1 second
• Prototypes and MVPs (add complexity later)
• Simple blogs, portfolios, basic CMSes

Complexity Threshold: Only worth it when hitting actual performance walls, not theoretical scaling concerns.

Docker Compose Production Template

version: '3.8'
services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: ${DB_NAME:-djangodb}
      POSTGRES_USER: ${DB_USER:-postgres}
      POSTGRES_PASSWORD: ${DB_PASSWORD:-postgres}
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER:-postgres}"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped

  redis:
    image: redis:7.4-alpine
    command: redis-server --appendonly yes --maxmemory 512mb --maxmemory-policy allkeys-lru
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 10s
      timeout: 3s
      retries: 5
    restart: unless-stopped

  web:
    build: .
    command: gunicorn core.wsgi:application --bind 0.0.0.0:8000 --workers 3
    environment:
      - DATABASE_URL=postgres://${DB_USER:-postgres}:${DB_PASSWORD:-postgres}@db:5432/${DB_NAME:-djangodb}
      - CELERY_BROKER_URL=redis://redis:6379/1
      - DEBUG=False
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_healthy
    restart: unless-stopped

  worker:
    build: .
    command: celery -A core worker --loglevel=info --concurrency=4 --max-tasks-per-child=1000
    environment:
      - DATABASE_URL=postgres://${DB_USER:-postgres}:${DB_PASSWORD:-postgres}@db:5432/${DB_NAME:-djangodb}
      - CELERY_BROKER_URL=redis://redis:6379/1
    depends_on:
      - db
      - redis
    restart: unless-stopped
    deploy:
      replicas: 2

volumes:
  postgres_data:
  redis_data:

Troubleshooting Decision Tree

1. Workers won't start: Check Docker service names in CELERY_BROKER_URL
2. Tasks execute synchronously: Verify CELERY_TASK_ALWAYS_EAGER = False
3. Tasks disappear: Enable Redis persistence (--appendonly yes)
4. Workers crash randomly: Set memory limits and max-tasks-per-child
5. Database connection errors: Increase max_connections, add connection pooling
6. Import errors: Ensure identical PYTHONPATH in all containers
7. Tasks stuck pending: Check queue routing vs worker queue configuration

Operational Intelligence Summary

This stack requires significant operational overhead but provides substantial performance benefits for applications processing heavy background work. The configuration complexity is front-loaded - once properly configured, it scales reliably. Most failures stem from Docker networking, memory management, and database connection limits rather than the core technologies. Investment worthwhile for applications with genuine background processing needs exceeding simple web request patterns.