Prometheus + Grafana + Jaeger: Production Observability Stack

Executive Summary

Purpose: Integrated observability stack for microservices debugging and monitoring
Implementation Time: 2-4 weeks for proper production deployment
Total Cost: $10K-50K annually (infrastructure only, no vendor surprises)
Operational Impact: Reduces debugging time from hours to minutes during outages

Critical Production Requirements

Resource Requirements (Actual Usage)

Memory Requirements (Critical - Plan Accordingly):

• Prometheus: 2GB RAM per 1 million active series
• Jaeger: 500MB RAM per collector instance
• Grafana: 512MB basic, 2GB+ for complex dashboards

Storage Reality Check:

• Metrics: 1-3 bytes per sample (manageable)
• Traces: 50KB average per trace (10-100x more storage than metrics)
• Budget 10x more storage for traces vs metrics

Performance Overhead:

• Prometheus: 1-3% CPU overhead for collection
• Jaeger: 2-5% CPU with proper sampling, 15%+ without sampling
• Application instrumentation: 1-2ms per operation

Scaling Thresholds and Breaking Points

Small Setup (< 10 services):

• 4GB RAM, 2 CPU cores, 100GB SSD
• Single instance deployment acceptable

Medium Setup (10-50 services):

• 16GB RAM, 4 CPU cores, 500GB SSD
• Requires clustering for availability

Large Setup (50+ services):

• 32GB+ RAM, 8+ CPU cores, 2TB+ SSD
• Mandatory federation/clustering, external storage

Critical Breaking Point: UI becomes unusable at 1000+ spans, making debugging large distributed transactions impossible

Implementation-Critical Configuration

Prometheus Production Settings

Essential Settings That Prevent Failures:

global:
  scrape_interval: 15s  # Don't go lower - kills disk performance
  evaluation_interval: 15s

# Critical flags for production
--storage.tsdb.retention.time=30d
--storage.tsdb.retention.size=50GB
--query.max-concurrency=10  # Prevents query-of-death scenarios

Common Configuration Failures:

• Port 9090 conflicts (use --web.listen-address=:9091)
• File descriptor limits cause random scrape failures (ulimit -n 65536)
• NTP drift breaks time-series queries with cryptic error messages

Jaeger V2 Critical Configuration

V2 vs V1 Migration Reality:

• V2 (November 2024): Built on OpenTelemetry, better performance
• Migration complexity: Moderate (configuration format changed)
• V1 compatibility issues with other tools resolved in V2

Sampling Strategy (Storage Cost Control):

processors:
  probabilistic_sampler:
    sampling_percentage: 5.0  # 5% base sampling
  tail_sampling:
    policies:
      - name: errors
        type: status_code
        status_code: {status_codes: [ERROR]}  # Always keep errors
      - name: slow_requests  
        type: latency
        latency: {threshold_ms: 2000}  # Always keep slow requests

Storage Backend Selection (Choose Carefully):

• Elasticsearch: Fast but unpredictable memory usage, expensive scaling
• Cassandra: Handles massive writes but requires specialized expertise
• ClickHouse: Best performance for new deployments, limited documentation

Critical Failure Modes and Solutions

Trace Collection Failures

Symptom: Traces not appearing in Grafana
Root Causes:

1. Jaeger collector connectivity: curl http://jaeger-query:16686/api/services returns connection refused
2. Wrong OTLP endpoint configuration in applications
3. Sampling set too aggressively (traces being dropped)

Solution Path:

1. Verify collector health endpoints
2. Check application OTLP exporter configuration
3. Temporarily increase sampling rate for debugging

Metrics Collection Failures

Symptom: Missing metrics in Prometheus
Root Causes:

1. Scrape targets failing with context deadline exceeded
2. /metrics endpoint returning HTML errors instead of Prometheus format
3. Port/path configuration mismatches

Solution Path:

1. Check Prometheus UI → Status → Targets for red entries
2. Validate /metrics endpoint format manually
3. Review scrape configuration for correct ports/paths

Query Performance Degradation

Breaking Points:

• Wide time ranges on high-cardinality metrics
• Regex operations on trace attributes
• Complex metric-trace correlation queries

Performance Optimization:

• Use recording rules for expensive PromQL aggregations
• Implement proper indexing in Jaeger storage backend
• Configure Grafana query caching with appropriate TTL

Operational Intelligence

Real-World Implementation Lessons

Time Investment Reality:

• Simple setup: 1 week (basic monitoring)
• Production-ready: 3-4 weeks (proper scaling, HA, integration)
• Team training: Additional 2-4 weeks for effective usage

Hidden Complexity Factors:

• Trace context propagation between services (often broken by default)
• Correlation ID management across service boundaries
• Alert tuning to avoid noise while catching real issues

Vendor Cost Comparison (Annual)

Solution	Implementation	Annual Cost	Vendor Lock-in	Best For
Prometheus/Grafana/Jaeger	2-4 weeks	$10K-50K	None	Teams wanting data ownership
DataDog APM	Days	$60K-400K+	High	Unlimited budget teams
New Relic	Days	$35K-250K+	High	Simple billing preference
Elastic Stack	4-8 weeks	$25K-180K	Medium	Elasticsearch expertise

Team Structure Requirements

Small Teams (< 20 engineers): Part-time person can manage
Medium Teams (20-100 engineers): Dedicated SRE + team champions
Large Teams (100+ engineers): Full observability team required

Critical Skills Needed:

• PromQL query optimization
• Distributed systems debugging
• Time-series database concepts
• Kubernetes/container orchestration

Production Deployment Patterns

High Availability Architecture

Prometheus HA:

• Requires federation or Thanos for true HA
• Single point of failure otherwise

Jaeger HA:

• Deploy collectors as DaemonSet in Kubernetes
• Use shared storage backend (Elasticsearch, Cassandra, ClickHouse)

Grafana HA:

• Stateless deployment behind load balancer
• External database for dashboard storage

Kubernetes Deployment Tools

Recommended Helm Charts:

• kube-prometheus-stack: Integrated Prometheus + Grafana + AlertManager
• jaeger-operator: Kubernetes-native Jaeger management
• OpenTelemetry Collector charts for trace collection

Troubleshooting Decision Trees

Trace-to-Metrics Correlation Issues

Problem: Cannot link metrics spikes to trace samples
Solution: Configure exemplars in Prometheus metrics with trace IDs

- name: http_request_duration
  exemplars:
    - trace_id="{{.trace_id}}"
      span_id="{{.span_id}}"

Resource Exhaustion Patterns

Memory Exhaustion Indicators:

• Prometheus: Query timeouts, OOM kills
• Jaeger: Collector restarts, trace drops
• Grafana: Dashboard loading failures

Immediate Actions:

1. Implement resource limits in Kubernetes
2. Configure retention policies
3. Enable trace sampling if not already active

Compliance and Security Considerations

Data Privacy (GDPR/PII)

Risk Areas:

• Traces can contain PII in span attributes
• Request payloads in trace data
• User identifiers in metric labels

Mitigation Strategies:

# Use OpenTelemetry redaction processor
processors:
  redaction:
    blocked_values: ["email", "ssn", "phone"]

Retention Policies:

• Implement automatic data deletion after compliance periods
• Use geographic data residency controls
• Implement role-based access to sensitive trace data

Success Metrics and KPIs

Implementation Success Indicators

Technical Metrics:

• Mean Time to Detection (MTTD): < 5 minutes for critical issues
• Mean Time to Resolution (MTTR): 50% reduction from baseline
• False positive alert rate: < 10%

Operational Metrics:

• Developer adoption: > 80% of services instrumented
• Dashboard usage: Active usage by all teams
• Query performance: 95th percentile < 5 seconds

Cost Optimization Strategies

Storage Optimization:

• Implement tiered retention (7 days hot, 30 days warm, 6+ months cold)
• Use sampling strategies to control trace volume
• Configure metric retention based on business requirements

Performance Optimization:

• Pre-compute expensive queries with recording rules
• Use read replicas for popular dashboards
• Implement caching for frequently accessed data

Critical Resources and Documentation

Essential References

• Prometheus: Official docs at prometheus.io/docs (configuration, querying, best practices)
• Jaeger: jaegertracing.io/docs (architecture, deployment, v2 migration)
• Grafana: grafana.com/docs (data sources, dashboards, alerting, exemplars)
• OpenTelemetry: opentelemetry.io/docs (SDKs, collector, instrumentation)

Production Examples

• Netflix: Processes 2+ trillion spans daily using this stack
• Uber: Billions of requests traced for ride debugging
• Shopify: Performance optimization through metric-trace correlation
• GitHub: Platform monitoring with Prometheus at scale

Community Support

• Prometheus: Google Groups forum (prometheus-users)
• Grafana: Official community forums (community.grafana.com)
• Jaeger: GitHub Discussions for troubleshooting
• OpenTelemetry: CNCF Slack (#opentelemetry channel)

Implementation Warnings

Will Break If:

• Trace sampling disabled (storage and performance death)
• File descriptor limits not increased (random connection failures)
• Time synchronization issues (cryptic query failures)
• Port conflicts ignored (Docker Desktop commonly conflicts)
• Memory limits not set (OOM kills during peak traffic)

Common Misconceptions:

• "Jaeger v1 and v2 are compatible" (configuration format completely changed)
• "100% trace sampling is fine for production" (will bankrupt storage budget)
• "Metrics and traces automatically correlate" (requires explicit configuration)
• "Default configurations work in production" (almost always need tuning)

This stack delivers complete observability when implemented correctly, but requires understanding these operational realities to avoid common pitfalls that waste weeks of debugging time.