Ollama Production Deployment: Critical Intelligence Summary

Critical Failure Modes and Solutions

Memory Management Disasters

SIGKILL after 10 minutes: OOMKiller terminating processes due to insufficient RAM

• Root cause: Model consumes more RAM than system can handle
• Real requirements: 24-32GB system RAM for "8GB" models (3-4x theoretical minimum)
• Breaking point: Multiple services + OS overhead + context window growth
• Solution: Set proper container memory limits, use smaller models

docker run -d --memory=16g --oom-kill-disable=false ollama/ollama

Memory leak pattern: Usage grows until crash, requires daily restarts

• Frequency: Affects certain models with long conversations
• Workaround: Daily restart automation
• Better fix: Clear conversation context in application code

Concurrency Bottlenecks

Response time degradation: 2 seconds → 45 seconds under load

• Root cause: Sequential request processing (OLLAMA_NUM_PARALLEL=1 default)
• Breaking point: >2 concurrent users
• Nuclear option: OLLAMA_NUM_PARALLEL=4 (multiplies VRAM usage)
• Production solution: Multiple Ollama instances with load balancer

Context contamination: Users receive other users' responses

• Severity: Critical security issue in production
• Immediate fix: Disable parallel processing
• Proper fix: Update to Ollama 0.11.5+ for better memory management

GPU Utilization Problems

Low GPU usage (20%): Memory fragmentation prevents efficient VRAM use

• Solution: Force specific GPU layers

OLLAMA_NUM_GPU_LAYERS=35 ollama run llama3.3:7b

Silent CPU fallback: GPU driver mismatches cause undetected failures

• Detection: Monitor nvidia-smi vs ollama ps differences
• Common causes: CUDA 11.8 vs 12.1 conflicts, suspend/resume cycles

Resource Requirements (Production Reality)

Memory Allocation Table

Model Size	Theoretical RAM	Production RAM Required	Max Concurrent Users
7B	8GB	24-32GB	10-20
13B	16GB	48-64GB	5-10
70B	40GB	120-160GB	1-3

Storage Performance Requirements

• Model loading time: 3+ minutes indicates storage bottleneck
• Minimum: Local NVMe SSD (1GB/s+ throughput)
• Avoid: Network storage (adds 30+ seconds), spinning disks
• Test command: dd if=/dev/zero of=/tmp/test bs=1G count=10 oflag=dsync

GPU Memory Multipliers

• Single instance: 1x model size
• OLLAMA_NUM_PARALLEL=8: 8x model size (320GB+ for 40GB model)
• Multiple instances: 1x per instance (horizontal scaling)

Production Configuration

Essential Environment Variables

export OLLAMA_NUM_PARALLEL=1      # Conservative start
export OLLAMA_KEEP_ALIVE=30m      # Balance memory vs response time
export OLLAMA_HOST=0.0.0.0        # External connections
export OLLAMA_ORIGINS="*"         # CORS configuration
export CUDA_VISIBLE_DEVICES=0     # Lock to specific GPU
export OLLAMA_DEBUG=1             # Production logging

Load Balancer Timeouts

Default timeouts (30s) cause failures

• NGINX: proxy_read_timeout 300s
• HAProxy: timeout server 300s
• Health check: GET /api/ps with 300s timeout

Container Resource Limits

deploy:
  resources:
    limits:
      memory: 24G
    reservations:
      memory: 16G

Deployment Architecture Comparison

Architecture	Max Users	Memory Overhead	Complexity	Failure Mode
Single Instance	5-10	Low	Easy	Any real load
Parallel Processing	10-20	High (VRAM×N)	Easy	GPU exhaustion
Multiple Instances	50-100	Medium	Medium	Load balancing
vLLM Migration	100+	Lower	High	API rewrite needed

Critical Monitoring Metrics

Real-time Failure Indicators

• Model load/unload frequency: High churn = memory pressure
• Response queue length: Sustained >3 = capacity needed
• GPU memory fragmentation: nvidia-smi vs ollama ps differences
• Context window sizes: Growth indicates memory leaks

Alert Thresholds

• Response time >10s for >5 minutes
• Queue depth >5 for >2 minutes
• GPU memory >90% for >30 seconds
• No successful responses for >5 minutes

Production Health Checks

Functional Health Check Script

#!/bin/bash
# Check Ollama response
curl -s HOST:11434/api/ps >/dev/null || exit 2

# Check GPU memory
GPU_MEM=$(nvidia-smi --query-gpu=memory.used --format=csv,noheader,nounits)
[ "$GPU_MEM" -gt 90 ] && exit 2

# Check system memory  
SYS_MEM=$(free | grep Mem | awk '{printf "%.0f", $3/$2 * 100}')
[ "$SYS_MEM" -gt 95 ] && exit 2

Migration Decision Matrix

When to Abandon Ollama

• >100 concurrent users: vLLM handles 793 TPS vs Ollama's 41 TPS
• Multiple simultaneous models: TGI has better resource management
• High availability requirements: Managed services for stability
• GPU utilization <30%: Resource waste indicates wrong tool

Migration Targets

• vLLM: Better concurrency, complex setup, API compatible
• TGI: Kubernetes-native, Hugging Face ecosystem
• Managed services: Higher cost, lower operational burden

Automated Recovery Implementation

Restart Script for Memory Leaks

while true; do
  sleep 60
  if ! /opt/ollama/healthcheck.sh; then
    pkill -f ollama
    nvidia-smi --gpu-reset
    systemctl restart ollama
    ollama run llama3.3:7b "warmup" >/dev/null 2>&1 &
  fi
done

Pre-production Load Testing

# Concurrent request test
for i in {1..10}; do
  curl -X POST HOST:11434/api/generate \
    -H "Content-Type: application/json" \
    -d '{"model": "llama3.3:7b", "prompt": "Test"}' &
done

Cost-Benefit Analysis

Resource Investment Reality

• Development: Works on MacBook (32GB unified memory)
• Production minimum: 3x theoretical requirements
• Operational overhead: Daily restarts, monitoring, debugging
• Break-even point: 50+ daily users before managed services cost less

Hidden Costs

• GPU driver maintenance: Silent failures require expertise
• Storage requirements: Local NVMe SSD mandatory
• Network configuration: Load balancer timeout tuning
• Monitoring setup: Custom metrics beyond standard monitoring

Critical Warnings

• Never use network storage for models in production
• Memory leaks require automated restarts - not optional
• GPU memory fragmentation causes performance degradation over time
• Default configurations fail under any realistic load
• Silent CPU fallback from GPU driver issues causes 10x slowdown
• Context contamination between users is a security vulnerability
• OOMKiller strikes without warning when memory planning inadequate