OpenTelemetry + Jaeger + Grafana on Kubernetes: Production Observability Stack

Stack Overview

Core Components:

• OpenTelemetry: Vendor-neutral instrumentation (CNCF project, v1.0+ stable since 2021)
• Jaeger v2: Distributed tracing storage and search (November 2024 release, built on OpenTelemetry Collector)
• Grafana: Visualization and dashboards (trace-to-metrics since 9.1, improved TraceQL in 11.0)
• Kubernetes: Container orchestration with service discovery

Key Value Proposition: Zero vendor lock-in, enterprise-scale performance (Netflix processes 2+ trillion spans daily), complete observability stack for free.

Critical Performance Specifications

Resource Requirements (Production Reality)

Component	Minimum RAM	Typical RAM	CPU	Storage Impact
OpenTelemetry Agent (DaemonSet)	150MB	300MB (spikes to 400MB)	0.1-0.2 cores	N/A
OpenTelemetry Gateway	500MB	1-4GB (can balloon to 8GB)	1-2 cores	N/A
Jaeger v2	500MB	1-3GB (storage dependent)	1-2 cores	Varies by backend
Grafana	250MB	400MB-2GB (dashboard complexity)	0.5-1 cores	Minimal

Performance Overhead

• Normal Operation: 2-5% CPU overhead
• Misconfigured: Up to 50% CPU overhead
• Memory: 50-200MB per collector (can leak to gigabytes)
• Network: Scales with trace volume

Critical Failure Modes

Silent Failures

• Collectors die silently when resource limits are exceeded
• Traces disappear without alerts when collectors fail
• Memory limits kill collectors without visible errors
• Default Helm chart configurations fail in production load

Configuration Hell

• Helm charts assume defaults work (they don't)
• Service mesh configs conflict with OpenTelemetry configs
• Storage backends timeout under production load
• Version 1.2.3 of OpenTelemetry operator has known memory leak in webhook

Production Breaking Points

• UI breaks at 1000+ spans making debugging impossible
• 100% sampling kills production - use 1-10% for high-traffic services
• Poorly configured collectors eat entire CPU cores
• Complex Grafana dashboards can consume 32GB RAM

Deployment Reality

Time Investment

• Optimistic: Few hours if nothing breaks
• Realistic: 2-3 weeks for production-ready deployment
• Disaster: Several weeks when everything breaks
• Expertise Required: Deep Kubernetes, YAML configuration, distributed systems knowledge

What Actually Breaks During Deployment

1. Resource limits too low (default charts)
2. Storage backend timeouts under load
3. Service mesh integration conflicts
4. Auto-instrumentation breaks authentication headers
5. Version compatibility issues between components
6. Network policies blocking component communication

Deployment Approaches Comparison

Method	Setup Time	Customization	Production Readiness	Maintenance
OpenTelemetry Operator	Fast	Limited (CRD hell for custom configs)	High	Medium
Helm Charts	Medium	YAML configuration nightmare	High	High
Manual Deployment	Slow	Complete control	Highest	Highest

Storage and Cost Reality

Storage Costs (Monthly)

• Budget Range: $500-5000/month depending on scale
• Retention Strategy: Detailed traces 7 days, aggregated metrics 6 months, trends forever
• Storage Backend Costs: ClickHouse (cheapest) < Cassandra < Elasticsearch (most expensive)
• Object Storage: Unlimited but slowest query performance

Sampling Strategy Requirements

• High-traffic services: 1-10% probabilistic sampling
• Error traces: Always retain via tail-based sampling
• Slow requests: Always retain via adaptive sampling
• Volume management: Essential to prevent collector death

Security Implementation

Required Security Measures

• mTLS for OTLP communications (security team requirement)
• Kubernetes network policies (default allow-all is dangerous)
• Grafana RBAC (prevent developers accessing billing dashboards)
• Data sanitization processors to remove sensitive information
• Cross-cluster service discovery configuration for multi-tenant deployments

Integration Complexity Matrix

Integration Type	Complexity	Performance Impact	Vendor Lock-in	Use Case
OpenTelemetry + Jaeger + Grafana	Moderate (3 weeks)	Low (2-5%)	None	Complete observability
Proprietary APM	Low (but expensive)	Medium (5-15%)	High	Budget > time
ELK Stack + APM	High (YAML nightmare)	Medium (3-10%)	Medium	Elasticsearch masochists
Cloud Provider Solutions	Low (until customization)	Variable (black box)	High	Cloud-native convenience

Critical Warnings and Operational Intelligence

What Documentation Doesn't Tell You

• Default configurations will fail in production load
• Collectors require health monitoring or failures go unnoticed
• Memory leaks are common in misconfigured deployments
• Service mesh integration requires understanding both Istio and OpenTelemetry configs
• Storage backends have different reliability characteristics under load

Migration Considerations

• Dual deployment strategy: Run alongside existing APM during transition
• Automatic instrumentation: Minimizes code changes but can break authentication
• Dashboard migration: Gradual transition while maintaining existing tooling
• Multi-cluster deployments: Require gateway aggregation points and centralized storage

Troubleshooting Hierarchy

1. Check collector logs first (usually the problem)
2. Verify OTLP endpoint reachability
3. Enable debug logging temporarily (fills disk quickly)
4. Confirm collectors are actually running (resource limits kill silently)
5. Monitor collector health religiously

Recommended Implementation Path

Phase 1: Foundation (Week 1)

• Deploy minimal OpenTelemetry Collector with basic configuration
• Set up Jaeger v2 with ClickHouse backend
• Configure basic Grafana dashboards
• Implement health monitoring for all components

Phase 2: Production Hardening (Week 2-3)

• Configure proper resource limits based on traffic patterns
• Implement sampling strategies (start with 5% probabilistic)
• Set up alerting for collector health and pipeline failures
• Configure security (mTLS, RBAC, network policies)

Phase 3: Scale and Optimize (Ongoing)

• Tune sampling rates based on production data
• Optimize storage retention policies
• Implement custom business metrics
• Monitor and adjust resource allocations

Success Criteria and Validation

Deployment Success Indicators

• Trace completeness: >95% of requests produce complete traces
• Collector uptime: >99.9% availability with automatic restart
• Query performance: Dashboard loads <10 seconds
• Resource stability: No OOM kills or CPU throttling
• Storage performance: Query response times <5 seconds

Common Failure Patterns to Monitor

• Trace volume spikes killing Elasticsearch
• Auto-instrumentation breaking application authentication
• Grafana memory consumption during complex dashboard rendering
• Collector resource exhaustion during traffic surges
• Storage backend timeouts during high query loads