OpenLIT Production Deployment: AI-Optimized Technical Reference

Critical Resource Requirements

ClickHouse (Primary Resource Constraint)

• Memory: 32GB minimum, 64GB for >5M traces/day
• Memory spikes: 5x during aggregations (major failure point)
• CPU: 8 cores minimum, 16 for heavy workloads
• Storage: 500GB SSD minimum, grows 10GB per million traces
• Network: 1Gbps between OTLP collector and ClickHouse

OTLP Collectors

• Latency-sensitive: 200-500ms cross-region penalty affects user experience
• Port conflicts: Default 4318 conflicts with Jaeger/other OpenTelemetry setups
• Recommended: Use port 4320 to avoid conflicts

Storage Growth Rate (Critical for Capacity Planning)

• 1M traces = ~10GB storage
• 100M traces/month = 1TB storage
• Indexes and aggregations add 30% overhead
• Failure scenario: 500GB disk filled in 3 days without retention policies

Production Cost Analysis

AWS Cost Structure (5M traces/day)

• 3x c5.2xlarge ClickHouse cluster: $1,200/month
• 2x c5.large OTLP collectors: $300/month
• 1TB EBS storage: $100/month
• Network transfer: $50-200/month
• Total: ~$1,650/month (cheaper than Datadog, self-managed)

Failure Modes and Solutions

Primary Failure Points (In Order of Frequency)

1. ClickHouse memory exhaustion → Memory limit exceeded errors

   • Solution: Set max_memory_usage = 20000000000 (20GB minimum)
   • Tune max_bytes_before_external_group_by for large aggregations

2. Disk space exhaustion → 90% of ClickHouse failures

   • Implement TTL policies from day one
   • Monitor storage growth: 10GB per million traces

3. OTLP collector buffering failure → Trace loss during restarts

   • Configure persistent queues with 5000+ queue_size
   • Enable retry_on_failure with exponential backoff

High Availability Requirements

• ClickHouse clustering: 3+ nodes minimum, requires ZooKeeper
• Implementation time: 2-3 days for proper setup
• Single points of failure: Complete observability loss if any component fails

Performance Thresholds

ClickHouse Performance Limits

• Query timeout: Set max_execution_time = 300 (5 minutes)
• Memory per query: max_memory_usage = 20000000000 (20GB)
• Thread limit: max_threads = 16 (match CPU cores)
• UI breaking point: >1M traces without time bounds kills performance

Scaling Breakpoints

Deployment Method	Scale Limit	Setup Time	Monthly Cost	Reliability
Docker Compose	~1M traces/day	15 minutes	$200	Single point of failure
Kubernetes Basic	~10M traces/day	2-4 hours	$800-1500	Good with replicas
Kubernetes HA	~100M traces/day	1-2 days	$2000-5000	Production ready
Managed Services	Unlimited	1 day	$3000-5000	Enterprise grade

Configuration That Actually Works

Production ClickHouse Settings

settings:
  max_memory_usage: 20000000000
  max_execution_time: 300
  max_threads: 16

OTLP Collector Buffering (Critical)

exporters:
  otlphttp:
    sending_queue:
      enabled: true
      num_consumers: 16
      queue_size: 5000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s

Storage Configuration

• Use GP3 over GP2: Better IOPS performance
• ReadWriteOnce only: ReadWriteMany corrupts ClickHouse data
• Backup strategy: No built-in backup, requires automated S3 snapshots

Security Critical Points

Data Exposure Risks

• Trace data contains secrets: LLM prompts/responses may include API keys, PII
• Default credentials vulnerability: Change user@openlit.io/openlituser immediately
• Network segmentation: Isolate ClickHouse from direct internet access

Required Filtering

processors:
  filter:
    traces:
      include:
        match_type: regexp
        attributes:
          - key: service.name
            value: "(openlit|llm-service)"

Downtime Expectations

Update Downtime Windows

• ClickHouse updates: 5-10 minutes (HA), 30+ minutes (single node)
• OTLP collector updates: Near zero with rolling deployments
• OpenLIT UI updates: 30 seconds, traces continue flowing
• Schema migrations: Hours on large trace tables

Disaster Recovery Times

• ClickHouse failure recovery: 30+ minutes
• OTLP collector recovery: Near-zero with proper setup
• Complete cluster failure: 2-4 hours (depends on data size)

Kubernetes Production Gotchas

Service Mesh Compatibility

• Istio: Requires PERMISSIVE mTLS mode for OTLP endpoints
• OTLP traffic: Doesn't work with strict mTLS

Load Balancer Requirements

• Session affinity: Must be OFF for OTLP collectors
• Health checks: Standard probes miss trace forwarding failures
• Use dedicated health check endpoint: Port 13133

Monitoring Requirements

Key metrics for alerting:

• otelcol_exporter_queue_size → traces backing up
• clickhouse_query_duration_seconds → database performance
• openlit_ui_response_time → dashboard availability

Debugging When OpenLIT is Down

External Monitoring Requirements

• Node exporter for host metrics
• Prometheus for ClickHouse metrics
• CloudWatch/Datadog for basic health checks

Emergency Query Access

-- Check trace ingestion in last hour
SELECT count() FROM otel_traces WHERE Timestamp > now() - INTERVAL 1 HOUR;

Retention and Compliance

Data Retention Implementation

-- Set 30-day retention
ALTER TABLE otel_traces 
MODIFY TTL toDate(Timestamp) + INTERVAL 30 DAY;

-- Add compression
ALTER TABLE otel_traces MODIFY COLUMN TraceId CODEC(LZ4HC);

GDPR Considerations

• Configure log_personal_data_max_bytes for data privacy
• Implement trace filtering for PII removal
• Document data retention policies

Resource References (Production-Tested)

Critical Documentation

• OpenLIT Helm Chart
• ClickHouse Performance Tuning
• OpenTelemetry Collector Scaling

Operational Tools

• ClickHouse Operator
• OpenTelemetry Operator
• Velero Kubernetes Backup

Community Support

• OpenLIT Slack Community
• GitHub Issues
• ClickHouse Slack