ChromaDB Production Troubleshooting Guide

Critical Version Information

• ChromaDB 1.0.21+: Fixed major memory leaks that required container restarts twice weekly
• ChromaDB 1.0.21 Breaking Change: Added AVX512 optimizations that crash on CPUs older than 2017
• Pre-1.0.21: Severe memory leaks, not production viable

Memory Management

Resource Requirements

• Formula: Collection size × 2.5 = minimum RAM required
• Production Reality: ChromaDB loads entire collection into memory
• Memory Leak Behavior: Memory usage only increases, never decreases
• Restart Trigger: When memory usage exceeds 80%

Memory Configuration

# Minimum production setup
docker run -d --memory=4g chromadb/chroma:latest

# High-performance setup
docker run -d --memory=8g --oom-kill-disable chromadb/chroma:latest

Critical Memory Errors

• bad allocation or std::bad_alloc: Out of memory crash
• Frequency: Most common production failure mode
• Impact: Complete service failure

Database Lock Issues

Common Lock Scenarios

• Multiple ChromaDB instances accessing same database
• Docker crashes leaving lock files behind
• Windows file system corruption (fundamental issue)

Resolution Steps

# Standard fix
find /path/to/chroma -name "*.lock" -delete

# Nuclear option
rm -rf /chroma_data && docker restart chromadb

Platform-Specific Issues

• Windows: File locking fundamentally broken, restart machine required
• WSL2: More reliable than native Windows
• Linux: Standard lock cleanup works

Permission Problems

Docker Volume Ownership

# Standard fix
sudo chown -R 1000:1000 /path/to/chroma_data
chmod 755 /path/to/chroma_data

# Nuclear option for persistent issues
chmod 777 /path/to/chroma_data

Breaking Conditions

• Username contains spaces (2-hour debugging time documented)
• Container user ID mismatches
• Kubernetes ownership changes

Dimension Mismatch Failures

Error Pattern

• ValueError: could not broadcast input array
• Cause: Embedding dimensions don't match collection expectations
• ChromaDB Limitation: No auto-conversion like Pinecone

Resolution (No Migration Path)

# Only solution: Complete recreation
client.delete_collection(collection_name)
collection = client.create_collection(collection_name)

AVX512 Compatibility Crisis

Impact

• Error: Illegal instruction (core dumped)
• Affected Versions: 1.0.21+
• CPU Requirement: Intel 2017+ or AMD equivalent

Detection and Workaround

# Check CPU support
cat /proc/cpuinfo | grep avx512
# If no output, must use 1.0.20

# Immediate fix
docker run -d chromadb/chroma:1.0.20

Performance Characteristics

Query Response Patterns

• Inconsistent Performance: Same query 10ms to 1000ms variation
• Memory Pressure Impact: Performance degrades with memory usage
• Default Embedding Model: Significantly slower than OpenAI ada-002

Performance Optimization

# Default (slow for production)
collection = client.create_collection("test")

# Production-grade (10x performance improvement documented)
openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="your-key",
    model_name="text-embedding-ada-002"
)
collection = client.create_collection("test", embedding_function=openai_ef)

Container Management

Health Check Limitations

• Reality: Health checks report success while returning 500 errors
• Verification Required: Manual endpoint testing necessary

Startup Failure Patterns (by frequency)

1. Port 8000 already in use
2. Mount path doesn't exist
3. Out of disk space (silent failure)

Debugging Commands

# Actual health verification
curl YOUR_HOST:8000/api/v1/heartbeat

# Container inspection
docker logs chromadb --tail 50
netstat -tuln | grep 8000

Critical Production Limits

Scale Boundaries

• Under 1M vectors: Usually stable
• 1-10M vectors: Requires active monitoring
• Over 10M vectors: Consider alternatives (Qdrant, Pinecone, pgvector)

Operational Requirements

• Weekly restarts: Memory usage creep mitigation
• Monitoring thresholds: Memory >80%, query time >500ms
• Backup requirements: Stop container before copying (SQLite corruption risk)

Error Classification Matrix

Error Type	Frequency	Resolution Time	Severity	Production Impact
Memory Exhaustion	High	5 minutes	Critical	Complete failure
SQLite Locks	Medium	2 minutes	High	Service unavailable
Permission Issues	Medium	30 seconds	Medium	Write failures
Dimension Mismatch	Low	10 minutes	High	Data loss (recreation required)
AVX512 Crashes	Low	1 minute	Critical	Complete failure
Network Timeouts	Medium	Variable	Medium	Partial failures

Production Readiness Assessment

Use ChromaDB When

• Vector count < 1M
• Team has debugging capacity
• Memory resources adequate (collection size × 2.5)
• Not latency-critical workloads

Consider Alternatives When

• Memory costs exceed budget
• Debugging time > development time
• Consistent performance required
• Multi-instance deployment needed

Migration Paths

• Qdrant: More stable, similar API
• Pinecone: Managed, expensive but reliable
• PostgreSQL + pgvector: Proven technology, fewer features

Essential Monitoring

Critical Metrics

# Memory trending
docker stats chromadb --format "table {{.MemUsage}} {{.MemPerc}}"

# Query latency
curl -w "@curl-format.txt" -s -o /dev/null YOUR_HOST:8000/api/v1/heartbeat

# Storage growth
du -sh /chroma_data

Alert Thresholds

• Memory usage > 80% (restart trigger)
• Query response > 500ms
• Disk growth > 1GB/day
• Container restart events

Concurrency Limitations

SQLite Constraints

• Poor concurrent write performance
• Lock contention under load
• No clustering/replication support

Workarounds

• Connection pooling where available
• Limit concurrent operations
• Separate instances with different data directories

Backup and Recovery

Backup Process

# Critical: Stop container first
docker stop chromadb
cp -r /your/chroma/data ./backup/
docker start chromadb

Recovery Considerations

• SQLite corruption during live backups
• No incremental backup support
• Backup verification essential before relying on them

Network and Deployment Issues

Common Production Failures

1. Memory limits: Production has less RAM than development
2. Network timeouts: Production networks slower/less reliable
3. File permissions: Production security stricter
4. Disk space: Production storage fills faster

Mitigation Strategies

• Test with production-like constraints locally
• Use wired connections for large imports
• Implement retry logic in application code
• Batch operations to 500-1000 vectors maximum