Jaeger: Distributed Tracing for Microservices - AI-Optimized Reference

Core Problem and Solution

Problem: Distributed systems debugging without visibility into request flow across services results in hours/days of blind troubleshooting
Solution: Jaeger provides distributed tracing showing complete request paths with timing and dependency information

Critical Production Realities

Storage Costs and Configuration

• Budget Impact: 100GB per million spans, costs $500-2000/month minimum for medium systems
• Default Configuration Failure: Elasticsearch defaults consume 64GB+ RAM and OOM nodes
• Production Settings Required:
  • Set retention policies immediately (7-day minimum)
  • Configure index lifecycle management for Elasticsearch
  • Implement adaptive sampling (start with 1% production sampling)

Operational Breaking Points

• UI Performance: Becomes unusable with 1000+ spans per trace, crashes browser tabs at 10k+ spans
• Collector Failure Modes:
  • Crashes on malformed spans or 50MB+ payloads
  • Default queue size insufficient for production load
  • Required settings: --collector.queue-size=100000 and --collector.num-workers=100
• Service Discovery Issues: Inconsistent service naming creates unusable dependency graphs

Architecture Decisions and Trade-offs

Jaeger v2 vs v1 (Critical Migration)

• v1 End-of-Life: December 31, 2025
• v2 Benefits: Eliminates agent deployment complexity, native OpenTelemetry support, single binary deployment
• Migration Impact: Agent-based deployments require complete reconfiguration

Storage Backend Selection

Backend	Pros	Cons	When to Use
Elasticsearch	Easier to operate, familiar	Expensive, memory-intensive	< 50 services
Cassandra	Massive scale capable	Requires specialized expertise	Large scale, dedicated team
ClickHouse	Faster, cheaper than ES	Harder to operate	Cost optimization priority

Failure Scenarios and Consequences

High-Impact Failures

• Collector OOM: Silently drops traces, creates debugging blind spots
• Storage Quota Exceeded: Complete tracing shutdown, no request visibility
• UI Timeout on Large Traces: Makes debugging complex distributed transactions impossible
• Sampling Misconfiguration: Either bankrupts storage budget or loses critical debugging data

Common Misconceptions

• "Tracing is safe to deploy": OpenTelemetry overhead can kill performance, requires canary deployments
• "Default settings work in production": All defaults are development-focused and will fail under load
• "Logs are equivalent to traces": Logs show service events, traces show request causality across services

Implementation Requirements

Prerequisites Not in Documentation

• DevOps expertise for Elasticsearch tuning
• Understanding of sampling strategies and storage mathematics
• Traditional monitoring for Jaeger components themselves
• Network partition and cluster split-brain recovery procedures

Resource Requirements

• Time Investment: Weekend initial setup, 0.5 FTE ongoing maintenance
• Technical Expertise: Elasticsearch operations, Kubernetes deployment, OpenTelemetry instrumentation
• Infrastructure: Separate cluster recommended for observability isolation

Performance Thresholds with Real-World Impact

• 1000+ spans per trace: UI becomes unusable for debugging
• 1M spans/day: Requires ~50GB storage/month minimum
• High-traffic services: Require 0.1% sampling or lower to control costs
• Elasticsearch heap: Must be tuned per production load or causes node failures

Decision Criteria for Alternatives

Choose Jaeger When:

• Running 10+ microservices with complex interactions
• Need cost control and infrastructure ownership
• Team has DevOps capacity for operational complexity
• OpenTelemetry standardization is priority

Choose Commercial APM When:

• < 50 services total
• Limited DevOps resources
• Budget allows $25+/host monthly costs
• Superior UX and alerting features required

Critical Warnings from Production Experience

What Official Documentation Omits:

• Storage costs scale exponentially with trace volume
• UI performance degrades significantly with complex traces
• Sampling configuration directly impacts both costs and debugging capability
• Collector resource limits must be set or system will OOM under load
• Service naming inconsistencies make dependency graphs unusable

Breaking Points and Thresholds:

• Traces > 1000 spans: UI performance issues
• Services > 100: Dependency graph becomes unreadable
• Sampling < 0.1%: May miss critical debugging traces
• Retention > 30 days: Storage costs become prohibitive for most teams

Configuration That Actually Works in Production

Collector Production Settings:

--collector.queue-size=100000
--collector.num-workers=100
--memory.max-traces=1000000

Sampling Strategy:

• Start: 1% production sampling
• High-traffic services: 0.1% or lower
• Critical paths: Force 100% sampling with custom logic
• Monitor: Adjust based on storage costs vs debugging needs

Storage Retention:

• Standard: 7-day retention minimum
• High-volume: 3-day retention with intelligent sampling
• Critical services: Extended retention for specific service subsets

Operational Intelligence Summary

Jaeger solves the distributed systems debugging problem effectively but requires significant operational investment. Success depends on proper sampling configuration, storage planning, and understanding performance limitations. The transition to v2 eliminates major deployment complexity but requires migration planning. Commercial alternatives may be more cost-effective for smaller deployments, but Jaeger provides superior cost control and flexibility at scale with appropriate DevOps investment.