OpenLIT Production Deployment - Battle-Tested Guide

The official docs say "minimal resources" which is complete bullshit. Here's what you actually need.

Resource Requirements That Don't Lie

ClickHouse is the resource hog. Plan accordingly:

• Memory: 32GB minimum, 64GB for >5M traces/day
• CPU: 8 cores minimum, 16 for heavy workloads
• Storage: 500GB SSD minimum, grows 10GB per million traces
• Network: 1Gbps between OTLP collector and ClickHouse

We crashed production twice before learning ClickHouse needs room to breathe. Memory usage spikes 5x during aggregations, especially when ingesting burst traffic from LLM workloads. The ClickHouse performance tuning guide covers memory optimization, while the OpenTelemetry collector scaling documentation explains resource planning. For production sizing, review the observability infrastructure requirements and ClickHouse cluster deployment guide. The Kubernetes resource management patterns show how to configure limits properly.

Network Architecture for Scale

OTLP endpoints are latency-sensitive. Keep collectors geographically close to your apps. Cross-region OTLP calls add 200-500ms to every request - your users will notice.

App Servers (US-East) → OTLP Collector (US-East) → ClickHouse (US-East)
App Servers (EU-West) → OTLP Collector (EU-West) → ClickHouse Replica

Port conflicts are real. Default 4318 conflicts with:

• Jaeger collectors
• Other OpenTelemetry setups
• Local development proxies

Pick custom ports and document them. We use 4320 for OpenLIT to avoid the mess.

Storage Strategy (Don't Fill Your Disks)

Trace retention grows faster than you think:

• 1M traces = ~10GB storage
• 100M traces/month = 1TB storage
• Indexes and aggregations add 30% overhead

Set up retention policies from day one. This killed our staging environment when traces filled a 500GB disk in 3 days.

-- ClickHouse retention policy example
ALTER TABLE otel_traces 
MODIFY TTL toDate(Timestamp) + INTERVAL 30 DAY;

High Availability Setup

Single points of failure that will bite you:

1. ClickHouse failure = complete observability loss
2. OTLP collector failure = trace ingestion stops
3. OpenLIT UI failure = dashboards go dark

Deploy everything in HA mode from the start. The official Helm chart supports replicas but doesn't configure persistent volumes properly. For production HA deployments, review the Kubernetes high availability patterns and ClickHouse replication strategies. The OpenTelemetry collector high availability guide covers load balancing approaches, while the observability stack resilience patterns explain disaster recovery procedures.

ClickHouse clustering is painful but necessary:

• 3+ nodes minimum for fault tolerance
• Shared storage or replication required
• ZooKeeper dependency adds complexity

Budget 2-3 days for proper ClickHouse clustering setup. The ClickHouse operator helps but brings its own operational overhead. For production clustering, follow the ClickHouse distributed table setup and ZooKeeper cluster configuration guide. The Kubernetes StatefulSet patterns show how to deploy clustered databases properly.

The official Helm chart gets you 80% there. Here's the other 20% that keeps you up at night.

Helm Values That Actually Work

## values-production.yaml
clickhouse:
  persistence:
    enabled: true
    size: 1Ti
    storageClass: gp3
  resources:
    requests:
      memory: 32Gi
      cpu: 8
    limits:
      memory: 64Gi
      cpu: 16
  settings:
    max_memory_usage: 20000000000
    max_execution_time: 300
    
collector:
  replicas: 3
  resources:
    requests:
      memory: 4Gi
      cpu: 2
    limits:
      memory: 8Gi
      cpu: 4
      
ui:
  replicas: 2
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
      cert-manager.io/cluster-issuer: letsencrypt-prod

Storage classes matter. GP3 volumes give you better IOPS than GP2. Don't cheap out on storage - ClickHouse performance depends on it. For production Helm deployments, follow the official Helm chart documentation and review the Kubernetes best practices guide. The ClickHouse Kubernetes operator provides advanced cluster management, while the OpenTelemetry Kubernetes operator handles auto-instrumentation.

Service Mesh Integration

Istio compatibility: OpenLIT works but requires specific config:

## Disable mTLS for OTLP endpoints
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: openlit-collector
spec:
  selector:
    matchLabels:
      app: openlit-collector
  mtls:
    mode: PERMISSIVE

OTLP traffic doesn't play nice with strict mTLS. We debugged this for 6 hours before finding the solution buried in GitHub issues.

Persistent Volume Gotchas

ClickHouse data corruption is real. Use ReadWriteOnce volumes, never ReadWriteMany. Shared storage corrupts ClickHouse's internal data structures.

persistence:
  storageClass: gp3
  accessMode: ReadWriteOnce
  size: 1Ti

Backup strategy is critical. ClickHouse doesn't have built-in backup. Set up automated snapshots:

## Daily backup script
kubectl exec clickhouse-0 -- clickhouse-client --query=\"BACKUP TABLE otel_traces TO S3('s3://backups/clickhouse/$(date +%Y%m%d)')\"

Load Balancer Configuration

OTLP collector needs session affinity OFF. Default round-robin works best for trace ingestion. Sticky sessions cause uneven load distribution.

service:
  type: LoadBalancer
  sessionAffinity: None
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: \"true\"

Health checks matter. OTLP collectors can appear healthy while failing to forward traces:

livenessProbe:
  httpGet:
    path: /
    port: 13133
  initialDelaySeconds: 30
readinessProbe:
  httpGet:
    path: /
    port: 13133
  initialDelaySeconds: 10

Monitoring Your Monitoring

Prometheus metrics for OpenLIT components:

## ServiceMonitor for collector metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: openlit-collector
spec:
  selector:
    matchLabels:
      app: openlit-collector
  endpoints:
  - port: metrics
    interval: 30s

Key metrics to alert on:

• otelcol_exporter_queue_size - traces backing up
• clickhouse_query_duration_seconds - database performance
• openlit_ui_response_time - dashboard availability

Disaster Recovery Procedures

When ClickHouse dies:

1. Check disk space first - 90% of failures
2. Restart the pod if memory exhausted
3. Restore from S3 backup if data corruption
4. Expect 30+ minutes recovery time

When OTLP collectors fail:

1. Rolling restart usually fixes it
2. Check persistent queue disk usage
3. Scale replicas if overwhelmed
4. Near-zero recovery time with proper setup

Complete cluster failure:

1. Restore ClickHouse data from S3
2. Deploy fresh Helm chart
3. Update OTLP endpoints in applications
4. Historical data preserved, new traces resume

Total recovery time: 2-4 hours depending on data size.