OpenTelemetry + Jaeger + Grafana on Kubernetes - The Stack That Actually Works

Microservices turned debugging into archaeology. One request touches 15 services, fails in the 12th one, and good luck figuring out why. Traditional monitoring tools shit the bed with this complexity.

I've spent years building observability stacks that work in production. Here's what I learned: you need OpenTelemetry for instrumentation (because vendor SDKs are trash), Jaeger for tracing (because following requests across services manually is hell), and Grafana for visualization (because readable dashboards matter).

What This Stack Actually Does

OpenTelemetry collects traces, metrics, and logs from your apps without tying you to any vendor. It's the CNCF's way of saying "fuck proprietary instrumentation." Version 1.0 dropped in 2021 and it's been solid since.

Jaeger stores and searches your traces. V2 just came out (November 2024) and it's built on OpenTelemetry Collector, which means native OTLP support and way less configuration hell. V2 supports ClickHouse, Elasticsearch, and Cassandra as storage backends, with gRPC and HTTP ingestion endpoints that actually work under load.

Grafana turns your data into dashboards people can actually read. Grafana's had trace-to-metrics since 9.1, but 11.0 improved the TraceQL integration, which is crucial when you're debugging cascading failures.

Kubernetes runs it all and provides service discovery that actually works (most of the time).

Why This Integration Doesn't Suck

Unlike most monitoring stacks that make you choose between vendor lock-in or configuration hell, this combination actually works:

OpenTelemetry standardizes everything - traces, metrics, logs all use the same format. No more vendor-specific data formats or proprietary SDKs that break every update.

Request tracing that actually works - W3C trace context flows through your entire stack. A request failing in service 12 gets tracked back to the originating API call, with timing for every hop.

Zero-code Kubernetes integration - The OpenTelemetry Operator injects sidecar collectors automatically. Your pods get instrumented without touching application code.

Netflix-scale for free - They process 2+ trillion spans daily using this architecture. That's enterprise scale without the enterprise price tag that makes your CFO cry.

The best part? It's all open source. No surprise licensing fees, no vendor account managers calling you every week, no \"contact sales for pricing\" bullshit.

Now that you understand why this stack makes sense, let's talk about actually deploying it. This is where good intentions meet production reality. Here's how to do it without losing your sanity.

Component Architecture That Won't Break

OpenTelemetry Collector runs two ways: agents on every node (DaemonSets) and central gateways (Deployments). Agents forward app data, gateways do the heavy lifting. Resource requirements: agents usually eat around 150MB, sometimes 300MB if you're unlucky, gateways can be anywhere from 500MB to "holy shit it's using 4GB again" (the docs say 50MB, they're wrong).

Jaeger v2 simplified everything by building on OpenTelemetry Collector. One binary, native OTLP, and storage backends that don't crash under load. Use ClickHouse for storage unless you enjoy paying Elasticsearch licensing fees.

Grafana needs maybe 250MB RAM for basic setups, maybe 2GB if you have data sources that timeout constantly. The unified alerting actually works now, unlike the old system that failed silently.

Kubernetes Integration Horror Stories

Operators vs. Helm vs. Manual Deployment:

• Operators: Work great until you need custom configs, then you're editing CRDs
• Helm: Fast setup but customization is YAML hell
• Manual: Takes forever but you understand what breaks

Resource Requirements (Real Numbers):

• OpenTelemetry Collector agents: usually around 150MB, sometimes spikes to 400MB when processing gets weird
• Gateway collectors: anywhere from 1GB to "why is this using 8GB" depending on traffic spikes
• Jaeger: 500MB minimum, but I've seen it balloon to 3GB (storage backend dependent)
• Grafana: starts at 400MB, grows to 1.5GB+ if your dashboards are complex

Security That Doesn't Break Everything:

• mTLS for OTLP (because security team demands it)
• Network policies (because default allow-all is stupid)
• RBAC for Grafana (because developers shouldn't see billing dashboards)

What Actually Breaks During Deployment

Everything. The Helm charts assume default configs work (they don't). Resource limits kill collectors silently. Service mesh configs conflict with OpenTelemetry configs. Storage backends timeout under load.

I've deployed this 12 times and here's what actually breaks: Version 1.2.3 of the operator broke our entire deployment because of a memory leak in the webhook, spent 6 hours debugging why collectors were dying - turned out to be memory limits set too low.

Plan for however long it takes - maybe a few hours if nothing breaks, maybe a week if everything does. And when collectors die, they die silently and your traces disappear. Monitor collector health religiously.