OpenTelemetry: AI-Optimized Technical Reference

Configuration That Actually Works in Production

SDK Setup by Language

Java

• Command: java -javaagent:opentelemetry-javaagent.jar -jar your-app.jar
• Performance Impact: 3-8% CPU overhead on 10k req/sec API
• Memory Overhead: ~50MB baseline
• Critical Failure: Breaks with custom classloaders and Spring Boot 3.2.x actuator endpoints
• Solution: Upgrade to Spring Boot 3.2.1+ or use manual instrumentation

Python

• Command: opentelemetry-bootstrap && opentelemetry-instrument python app.py
• Performance Impact: +23ms latency (15ms → 38ms baseline)
• Memory Overhead: ~30MB
• Critical Failure: Auto-instrumentation conflicts with gevent
• Solution: Use manual instrumentation for gevent applications

Node.js

• Status: Unreliable auto-instrumentation
• Critical Failure: ESM modules break auto-instrumentation completely
• Solution: Use CommonJS or manual instrumentation
• Working Components: Express instrumentation is stable

Go

• Approach: Manual instrumentation only
• Advantage: Clean, predictable API
• Trade-off: More development overhead

Collector Deployment Patterns

Mode	RAM Usage	Failure Mode	Use Case
Sidecar	200MB per pod	Pod resource exhaustion	Low-latency requirements
Gateway	Shared resources	Single point of failure	Cost optimization
Agent	Per-node baseline	CNI networking issues	Balanced approach

Critical Configuration

processors:
  memory_limiter:
    limit_mib: 512
  batch:
    timeout: 1s
    send_batch_size: 1024

Sampling Configuration

Production Settings

• Start: 1% sampling (trace_id_ratio_based: 0.01)
• Storage Cost: 2TB traces/month for 50-service architecture = $500/month S3
• Critical Warning: Head-based sampling may miss important errors

Resource Requirements

Time Investment

• Setup Time Reality: 1-2 weeks (not "2-3 days" as marketed)
• Learning Curve: Steep - plan for 3-4 weeks team ramp-up
• Operational Overhead: Significant for self-hosted solutions

Cost Structure

Component	Monthly Cost	Scale Factor
Self-hosted (Jaeger + Prometheus)	$200-1k	Infrastructure + engineer time
Grafana Cloud	~$300/month	10GB/day ingestion
Commercial APM	$15k-50k+	High traffic penalty
OpenTelemetry framework	$0	Storage costs apply

Performance Impact Thresholds

• Acceptable: 1-5% CPU overhead
• High-frequency operations: 15% performance hit with 1000+ calls/request
• Memory baseline: 200MB collector + 50-100MB per 1k spans/sec

Critical Warnings

Collector Memory Leaks

• Affected Version: 0.89.0 has memory leak with tail sampling processor
• Fix: Upgrade to 0.90.0+ immediately
• Monitoring: Always configure memory_limiter processor

High Cardinality Metrics

• Failure Scenario: User IDs as labels → 2M unique time series → 500GB Prometheus storage
• Prevention: Remove user IDs, request IDs, timestamps from metric labels
• Solution: Design for aggregation, not individual tracking

Network Timeout Issues

• Default Timeout: 10 seconds (insufficient for production)
• Fix: Set 30+ second timeouts
  • Java: otel.exporter.otlp.timeout=30000
  • Python: OTEL_EXPORTER_OTLP_TIMEOUT=30000

Context Propagation Failures

• Root Cause: Missing spans due to broken trace context
• Debug: Enable collector debug logging (service.telemetry.logs.level: debug)
• Warning: Debug logging generates excessive output

Decision Criteria

Choose OpenTelemetry When

• Vendor lock-in is unacceptable
• Multi-language environment (20+ supported languages)
• Team can handle operational complexity
• Long-term cost control is priority

Avoid OpenTelemetry When

• Team lacks distributed systems expertise
• Need immediate production deployment
• Simple monolith application
• Budget allows commercial APM without vendor concerns

Self-hosted vs Commercial Backends

Self-hosted (Jaeger + Prometheus)

• Pros: Total control, predictable costs
• Cons: Operational burden, capacity planning required
• Expertise Required: Kubernetes, storage optimization, performance tuning

Commercial Backends

• Pros: Managed infrastructure, support
• Cons: Vendor lock-in risk, cost scaling issues
• Best Options: Grafana Cloud (reasonable pricing), AWS X-Ray (native AWS integration)

Breaking Points and Failure Modes

Known Version Issues

• Spring Boot 3.2.0: Breaks custom actuator endpoints
• Collector 0.89.0: Memory leak in tail sampling processor
• Node.js ESM: Auto-instrumentation completely broken

Production Gotchas

• Kubernetes CNI: Host networking breaks in some configurations
• Resource Limits: Development configs fail in production resource constraints
• Service Mesh: Sidecars can interfere with collector networking

Debugging Missing Spans

1. Check sampling: 99% of missing spans are due to sampling configuration
2. Verify timeouts: Network issues between app and collector
3. Examine context: Trace context broken in service chain
4. Monitor collector: Overwhelmed collectors drop data silently

Implementation Reality

Semantic Conventions Status (September 2025)

• HTTP spans: Stable and adopted
• Database operations: Stabilized in 2025
• RPC calls: Still unstable despite roadmaps
• Legacy compatibility: Expect http.method and http.request.method coexistence

Community Support Quality

• GitHub Issues: Well-documented, active maintainer response
• Slack Community: Active support, maintainers respond
• Documentation: Improving but has gaps, Stack Overflow often required

Update Risk Management

• Rule: Never update on Friday/Monday
• Reality: Something will break with updates
• Strategy: Staged rollouts, quick rollback capability

Vendor Ecosystem (90+ Options)

Reliable Backends

• Jaeger + Prometheus: Self-hosted standard
• Grafana Cloud: Managed, reasonable pricing until ingestion limits
• AWS X-Ray: Native AWS support, confusing sampling rules
• Elastic APM: Good for log-heavy workloads

Integration Quality

• Data ingestion: Most vendors support OTLP
• Feature parity: Varies significantly between vendors
• Migration: OpenTelemetry enables backend switching without code changes

Success Metrics

Technical KPIs

• Trace capture rate: >95% of critical transactions
• Query performance: <2 second trace lookup
• Resource overhead: <5% application performance impact
• Storage efficiency: Controlled cardinality metrics

Business Value

• MTTR reduction: Faster incident resolution
• Vendor flexibility: Backend switching capability
• Cost predictability: Controlled observability spend
• Engineering efficiency: Standardized instrumentation across services