Prometheus + Grafana + Jaeger: Stop Debugging Microservices Like It's 2015

Most teams are flying blind with their microservices. You've got Prometheus scraping metrics, Jaeger collecting traces, and Grafana showing pretty charts — but when everything goes to hell, you're still clicking between fifteen different tabs trying to figure out what broke.

Spent 4 hours last month debugging an API that suddenly started taking forever. CPU was spiking, database looked fine, but couldn't connect the dots. Turns out a poorly indexed query was choking the whole system, but it took way too long to figure that out because metrics and traces weren't talking to each other. Our monitoring shit the bed during a flash sale, we lost a bunch of revenue, and found out from angry tweets. This integration fixes that clusterfuck.

Why Microservices Monitoring Is A Nightmare

Remember when you had one big app on one server? CPU goes up, you check the slow queries, problem solved. Those days are dead.

Now your "simple" login request bounces through 12 different services running on who-knows-how-many containers. When everything goes to hell, you're stuck asking:

• Which fucking service is the bottleneck?
• What specific code path decided to shit the bed?
• How many other services are now choking because of this?
• When did this dumpster fire actually start?

Netflix processes 2+ trillion spans daily (yeah, trillion with a T) just to keep their platform working. Uber traces billions of requests because they learned the hard way that debugging ride requests across thousands of services is impossible without it. They didn't build this for fun — they built it because distributed systems will drive you insane without proper tracing.

What Each Tool Actually Does (When It's Not Broken)

Prometheus is the least painful way to collect metrics. It pulls data from /metrics endpoints every 15 seconds and stores the numbers in a way that doesn't make you want to quit programming. PromQL (the query language) sucks at first but gets tolerable once you memorize the 12 functions that actually matter.

Jaeger tracks requests as they bounce around your microservices disaster. The v2 release actually doesn't suck — they rebuilt it on OpenTelemetry so it stops fighting with every other tool in your stack.

Grafana makes pretty pictures from your ugly data. More importantly, recent versions let you click from "this trace is slow as shit" directly to "here's why your database is crying." That click-through is the only reason this whole integration is worth the pain.

How to Actually Make This Shit Work Together

The only way this setup doesn't waste your time is if the tools actually talk to each other. Shopify figured this out when they were hunting down performance issues, and Airbnb uses similar patterns to keep their platform from falling over:

1. Application Instrumentation Layer

• Applications expose Prometheus metrics via /metrics endpoints
• The same applications generate distributed traces using OpenTelemetry SDKs
• Traces include trace IDs that can be correlated with metric labels

2. Collection and Storage Layer

• Prometheus scrapes metrics and stores them in its time-series database
• Jaeger collects traces via OTLP (OpenTelemetry Protocol) and stores them in Elasticsearch, Cassandra, or ClickHouse
• Both systems retain data based on business requirements (typically 7-30 days for traces, 6+ months for metrics)

3. Correlation and Visualization Layer

• Grafana connects to both Prometheus and Jaeger as data sources
• Dashboards show metrics with embedded trace queries
• Alert rules can trigger on metrics and include trace context in notifications

4. Unified Query Interface

• PromQL for metric aggregation and analysis
• TraceQL for trace filtering and searching
• LogQL if using Loki for logs (optional but recommended)

Why Not Just Pay DataDog $50K/Month?

Look, DataDog works great until you get the bill. They start at $15/host and then bend you over with usage charges. New Relic does the same shit but charges per GB ingested. I've seen companies get $80K monthly bills because someone left debug logging on.

This open-source stack costs you time and sanity upfront, but:

• You own your fucking data — no vendor can hold it hostage
• No artificial limits on what you can measure
• Keep data as long as you want without bankruptcy
• Actually understand how your monitoring works (helpful when it breaks during holiday weekend deployments)

Grafana's trace-to-metrics correlation lets you click from a slow trace directly to the related metric spikes, saving hours of detective work during outages.

Why This Actually Saves Your Ass During Outages

Here's where this integration pays for itself: when your API shits the bed during the company all-hands, Grafana's alerting doesn't just scream "SOMETHING IS BROKEN." It gives you:

• Metric context: "Response times went to shit - like 200ms jumped to over a second"
• Trace samples: Links to the exact slow requests causing the problem
• Timeline: "Started about 15 minutes ago, looks like it's hitting a bunch of users"

Instead of spending an hour playing detective, you click one link and see the database query that's choking. Google's SRE people figured this out years ago — alerts that tell you what to fix, not just that something is fucked.

Don't Be a Hero - Plan For Scale From Day One

This stack scales if you're not an idiot about it. Here's the math you need:

Memory Requirements (Don't Ignore These):

• Prometheus: 2GB RAM per million active series (yes, million)
• Jaeger: 500MB RAM per collector (deploy more collectors, not bigger ones)
• Grafana: 512MB for basic dashboards, 2GB+ if someone went nuts with queries

Storage Reality Check:

• Traces are 10-100x fatter than metrics
• Use sampling or go bankrupt (1-10% is normal)
• S3/GCS for long-term storage unless you like buying disks

High Availability (Because Shit Breaks):

• Prometheus needs federation for HA (pain in the ass but necessary)
• Jaeger clusters work with shared storage backends
• Grafana is stateless — stick it behind a load balancer and call it done

Netflix runs this exact setup at Netflix scale. GitHub monitors their entire platform with Prometheus. If it's good enough for them, it's probably good enough for your startup's 3 microservices.

Setting up this integration is where good intentions meet production reality. After several deployments, here's what actually works and what will fuck up your weekend if you ignore it.

Prerequisites and Planning (Skip This and Suffer Later)

Before touching any configs, figure out your scale or you'll regret it. Learned this the hard way when a "simple" deployment ate way more RAM than expected because I didn't do the math on trace volume. Don't make the same mistake.

Capacity Planning (Use These Numbers):

• Small setup (< 10 services): 4GB RAM, 2 CPU cores, 100GB SSD
• Medium setup (10-50 services): 16GB RAM, 4 CPU cores, 500GB SSD
• Large setup (50+ services): 32GB+ RAM, 8+ CPU cores, 2TB+ SSD

Network Requirements:

• Prometheus scrapes every 15 seconds by default (configurable)
• Jaeger trace ingestion can spike to 10x normal during load tests
• Grafana needs low-latency access to both data sources

Storage Math That Matters:

• Prometheus: ~1-3 bytes per sample, retention configurable
• Jaeger: ~50KB average per trace, depends heavily on span count
• Budget 10x more storage for traces than metrics

Step 1: Prometheus Setup That Won't Die Under Load

Start with a recent Prometheus version because older versions have memory leaks that will ruin your weekend. Been burned by this before. The official installation guide tells you how to install it, but here's the config that actually works in production:

## prometheus.yml - The config that actually works
global:
  scrape_interval: 15s     # Don't go lower unless you hate your disk
  evaluation_interval: 15s
  external_labels:
    cluster: 'production'  # Useful for federation later

rule_files:
  - "alert_rules/*.yml"
  - "recording_rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - alertmanager:9093

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'application-metrics'
    static_configs:
      - targets: ['app1:8080', 'app2:8080', 'app3:8080']
    scrape_interval: 10s
    scrape_timeout: 8s
    metrics_path: /metrics
    params:
      format: ['prometheus']

  - job_name: 'jaeger-metrics'
    static_configs:
      - targets: ['jaeger-collector:14269']
    scrape_interval: 30s

Settings That Will Save Your Ass:

• Memory settings: Use --storage.tsdb.retention.time=30d and --storage.tsdb.retention.size=50GB or watch your disk fill up and die
• Query limits: Set --query.max-concurrency=10 because some idiot will write a query that kills your server
• Remote storage: Thanos or Cortex if you want long-term retention without buying more disks

Shit That Will Break (And How I Know):

• Port conflicts: 9090 is always taken by some random service, use --web.listen-address=:9091. Docker Desktop randomly decides to own every port.
• File descriptor limits: Bump ulimit -n 65536 or random scrapes fail with dial tcp: too many open files. Always happens during demos.
• Time sync: NTP drift fucks up time-series queries with cryptic query processing would load too many samples errors

Step 2: Jaeger Deployment (V2 Architecture)

Jaeger v2 completely changed everything. If you're still running v1, upgrade when you can — the migration is worth it. V1 fought with everything in our stack, v2 actually plays nice. V2 uses OpenTelemetry Collector under the hood, so it stops being a pain in the ass.

## jaeger-v2-config.yml - Production ready configuration
extensions:
  healthcheck:
    endpoint: 0.0.0.0:13133

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 1s
    send_batch_size: 8192
  
  probabilistic_sampler:
    sampling_percentage: 5.0  # 5% sampling - adjust based on volume

exporters:
  jaeger:
    endpoint: jaeger-collector:14250
    tls:
      insecure: true

service:
  extensions: [healthcheck]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, probabilistic_sampler]
      exporters: [jaeger]
  telemetry:
    metrics:
      address: 0.0.0.0:8888

Storage Backend Selection (Choose Wisely):

• Elasticsearch: Blazing fast until it randomly OOMs during peak traffic, costs more than your car payment, and the tuning docs were written by someone who clearly hates you
• Cassandra: Handles insane write volume like a champ but you need a Cassandra wizard on staff (spoiler: they're all at Netflix now)
• ClickHouse: New hotness with sick performance, but the day you hit an edge case is the day you realize half the docs don't exist yet
• Memory: Only for dev/testing unless you enjoy losing data during the first restart

ClickHouse seems to be the sweet spot for new deployments — until you hit an edge case and realize the docs are still being written.

Oh, and another thing: whoever decided that every database needs its own query language should be forced to debug connection pooling issues for eternity. But I digress.

Step 3: Application Instrumentation (Where Dreams Go to Die)

This is where most teams give up and just buy DataDog. Your apps need to expose Prometheus metrics AND generate traces, and doing both without breaking everything requires some planning. OpenTelemetry SDKs handle both and save you from writing custom instrumentation.

Go Example (Most Common in Microservices):

import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/exporters/prometheus"
    "go.opentelemetry.io/otel/sdk/metric"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initObservability() {
    // Trace exporter to Jaeger
    traceExporter, _ := otlptracehttp.New(
        context.Background(),
        otlptracehttp.WithEndpoint("http://jaeger-collector:4318/v1/traces"),
    )
    
    // Metrics exporter to Prometheus
    metricExporter, _ := prometheus.New()
    
    // Set up both providers
    tp := trace.NewTracerProvider(trace.WithBatcher(traceExporter))
    mp := metric.NewMeterProvider(metric.WithReader(metricExporter))
    
    otel.SetTracerProvider(tp)
    otel.SetMeterProvider(mp)
}

Python Flask Example:

from opentelemetry import trace, metrics
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.prometheus import PrometheusMetricReader
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.metrics import MeterProvider

## Initialize telemetry
trace.set_tracer_provider(TracerProvider())
metrics.set_meter_provider(MeterProvider(readers=[PrometheusMetricReader()]))

## Instrument Flask automatically
FlaskInstrumentor().instrument_app(app)

## Add OTLP trace export
otlp_exporter = OTLPSpanExporter(endpoint="http://jaeger-collector:4318/v1/traces")
span_processor = BatchSpanProcessor(otlp_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

Things You Learn the Hard Way:

1. Trace Context Propagation: Make sure W3C Trace Context headers actually flow between services (they won't by default)
2. Correlation IDs: Put trace IDs in your logs or you'll be correlating traces manually
3. Sampling Strategy: Start with low sampling, crank it up when debugging, dial it back when storage costs get ugly
4. Error Tracking: Always set span status on errors: span.SetStatus(codes.Error, "Database timeout") - saves debugging time later

Step 4: Grafana - Making Your Data Less Ugly

Recent Grafana versions actually have decent trace-to-metrics correlation now. Setting it up is easy — building dashboards that don't make your team want to quit is the hard part.

Data Source Configuration:

## grafana/provisioning/datasources/datasources.yml
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    
  - name: Jaeger
    type: jaeger
    access: proxy
    url: http://jaeger-query:16686
    uid: jaeger-ds
    
  - name: Loki  # Optional but recommended
    type: loki
    access: proxy
    url: http://loki:3100

Dashboard Patterns That Actually Work:

1. Application Overview Dashboard:

   • Request rate, error rate, latency (RED metrics)
   • Links to trace examples for each service
   • Resource utilization (CPU, memory, connections)

2. Service-Specific Dashboards:

   • Endpoint-level metrics with trace correlation
   • Database query performance with sample traces
   • External dependency health

3. Infrastructure Dashboard:

   • Node metrics, container health
   • Storage utilization for Prometheus and Jaeger
   • Network performance between services

Trace-to-Metrics Correlation Setup:

{
  "targets": [
    {
      "expr": "histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))",
      "exemplar": true,
      "format": "time_series"
    }
  ],
  "options": {
    "exemplars": {
      "datasource": {
        "type": "jaeger",
        "uid": "jaeger-ds"
      }
    }
  }
}

Step 5: Production Deployment Patterns

Docker Compose for Development:
Use the official Prometheus Docker Compose example as a starting point, but add Jaeger and configure data source integration.

Kubernetes Production Deployment:

• kube-prometheus-stack: Helm chart with Prometheus, Grafana, AlertManager
• Jaeger Operator: Kubernetes-native Jaeger deployment
• Service mesh integration: Works with Istio and Linkerd

High Availability Considerations:

• Prometheus: Use federation or Thanos for HA
• Jaeger: Deploy collectors as DaemonSet, use shared storage backend
• Grafana: Stateless deployment behind load balancer, use external database

Troubleshooting Common Integration Failures

Traces Not Appearing in Grafana:

1. Verify Jaeger data source connectivity: curl http://jaeger-query:16686/api/services - usually returns connection refused because you got the port wrong
2. Check trace ingestion: Monitor Jaeger collector metrics in Prometheus (if they're not also broken)
3. Validate application instrumentation: Look for failed to export span errors in application logs - the OTLP endpoint is probably down again

Metrics Missing from Prometheus:

1. Check scrape targets: Prometheus UI → Status → Targets - half of them will be red with context deadline exceeded
2. Verify /metrics endpoint format: Should follow Prometheus exposition format but usually returns HTML error pages instead
3. Review scrape configuration: Ensure correct ports and paths - someone changed the port and didn't tell you

Poor Query Performance:

1. Optimize PromQL queries: Use recording rules for expensive aggregations
2. Tune Jaeger storage: Adjust retention and configure proper indexes
3. Scale Grafana: Use caching and read replicas for dashboards

Resource Consumption Issues:

1. Implement Prometheus recording rules to pre-compute expensive queries
2. Tune Jaeger sampling: Use adaptive sampling to maintain consistent trace volume
3. Configure retention policies: Balance storage costs with debugging requirements

This setup works when you don't cut corners. Start simple — get Prometheus working, then Jaeger, then connect them through Grafana. Don't try to do everything at once or you'll spend weeks debugging integration issues that could have been avoided.

Took me 3 hours last time just to figure out why traces weren't showing up (the collector was running but not actually collecting anything because of a config typo). My boss kept asking for ETAs while I'm staring at empty dashboards. Fun times.

Plan a few weeks for proper implementation - longer if your team has never done observability before.