OpenTelemetry Collector: AI-Optimized Technical Reference

Overview

Purpose: Telemetry data proxy that sits between applications and monitoring backends, enabling vendor independence and cost control.

Critical Success Story: Reduced observability costs by 73% (from $18k/month to ~$5k/month) by routing 90% of data to cheaper backends while keeping critical alerts in expensive vendor.

Architecture Components

Pipeline Structure

• Receivers: Accept data in various formats (OTLP, Jaeger, Zipkin, Prometheus)
• Processors: Transform, sample, filter, and enrich data
• Exporters: Ship processed data to backends

Deployment Patterns

Pattern	Use Case	Resource Impact	Failure Mode
Agent (sidecar/daemon)	Critical services	Higher resources	Independent failures
Gateway (centralized)	Non-critical services	Lower resources	Single point of failure

Configuration Requirements

Production-Critical Memory Settings

processors:
  memory_limiter:
    limit_mib: 1024        # REQUIRED: Without this, OOM killer terminates collector
    spike_limit_mib: 256   # Prevents memory spikes
    check_interval: 1s     # Monitor frequently under load

Failure Impact: Without memory limiter, collector consumes all available RAM and crashes entire node.

Minimal Working Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:         # MUST be first processor
    limit_mib: 1024
  batch:
    send_batch_size: 1024 # Conservative starting point
    timeout: 1s

exporters:
  otlp/backend:
    endpoint: backend:4317
    tls:
      insecure: true      # Use TLS in production

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/backend]
  telemetry:
    metrics:
      address: 0.0.0.0:8888  # Enable for debugging

Resource Planning

Memory Requirements (Real-World)

• Baseline: 512MB minimum (official 200MB is insufficient)
• Scaling: +200MB per 1K spans/second with tail sampling
• High cardinality: Double baseline for metrics with many dimensions
• Contrib components: Add 50% overhead

Performance Thresholds

• UI breakdown: >1000 spans causes debugging interface failure
• Queue backup: >1000 items indicates impending failure
• Batch sizing: 1024-8192 optimal, smaller causes CPU overhead

Critical Warnings

Version Blacklist

• v0.89.0: Memory leaks causing frequent crashes
• v0.82.x: Performance degradation under high throughput
• v0.78.2: Batch processor reliability issues
• Any .0 release: Wait for .1 or .2 patches

Common Production Failures

OOM Killer Termination

Symptoms: signal: killed in logs, no collector output
Root Cause: No memory limiter configured
Detection: dmesg | grep "killed process.*otelcol"
Prevention: Always configure memory_limiter as first processor

Data Loss During Crashes

Cause: No persistent queues configured
Impact: 4+ hours of telemetry data lost during outages
Solution: Enable file_storage extension with persistent queues

High Cardinality Metrics Explosion

Symptoms: Storage grows from 100GB to 2TB overnight
Cause: User IDs, session IDs, or request IDs in metric labels
Cost Impact: Can bankrupt observability budget within days
Mitigation: Filter high-cardinality labels before export

Cost Optimization Strategies

Tail Sampling Configuration

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors_always
        type: status_code
        status_code: {status_codes: [ERROR]}
      - name: sample_normal
        type: probabilistic
        probabilistic: {sampling_percentage: 5}  # 95% cost reduction for normal traces

Impact: Samples complete traces rather than random spans, maintains debugging capability while reducing costs.

PII Removal (Security + Compliance)

processors:
  attributes:
    actions:
      - key: user.email
        action: delete
      - key: credit_card.*
        action: delete
      - key: http.request.body    # Often contains sensitive data
        action: delete

Production Operations

Essential Monitoring Alerts

# Memory usage climbing toward limit
otelcol_process_memory_rss_bytes > 1GB

# Data flow stoppage
rate(otelcol_receiver_accepted_spans_total[5m]) == 0

# Export failures indicating backend issues
rate(otelcol_exporter_send_failed_spans_total[5m]) > 0

# Queue backup indicating performance degradation
otelcol_exporter_queue_size > 1000

Troubleshooting Commands

# Test OTLP HTTP endpoint
curl -X POST http://localhost:4318/v1/traces -d '{"traces":[]}'

# Get CPU profile for performance debugging
curl -o cpu.prof http://localhost:8888/debug/pprof/profile?seconds=30

# Check for OOM killer activity
dmesg | grep "killed process.*otelcol"

# Verify collector is listening
netstat -tulpn | grep :4317

Vendor Comparison Matrix

Aspect	OpenTelemetry Collector	Direct SDK Export	Vendor Agents
Vendor Lock-in Risk	None - change backends via config	High - code changes required	Extreme - vendor dependent
Multi-backend Support	Native - send to 3+ simultaneously	None - single destination	None - vendor specific
Cost Control	High - filter/sample before export	Low - pay for all generated data	None - vendor pricing control
Setup Complexity	Medium - YAML configuration	Low - until backend change needed	Low - until vendor changes
Resource Usage	~500MB RAM realistic	<50MB per service	100-300MB + vendor overhead
Data Processing	Advanced - sampling, filtering, PII removal	Basic - batching only	Vendor limited
Network Security	Single egress point	Multiple service endpoints	Vendor specific requirements

Implementation Decision Tree

When to Use Collector

• Multi-vendor strategy: Planning to use multiple backends
• Cost optimization: Need to filter/sample before paying for data
• Security requirements: PII removal or data transformation needed
• Vendor independence: Avoiding lock-in to specific observability provider

When Direct Export Acceptable

• Single vendor commitment: Long-term contract with trusted provider
• Simple requirements: No data transformation needed
• Resource constraints: Cannot spare 500MB RAM for collector
• Rapid prototyping: Speed over flexibility

Security Requirements

Production TLS Configuration

receivers:
  otlp:
    protocols:
      grpc:
        tls:
          cert_file: /etc/ssl/certs/otel.crt
          key_file: /etc/ssl/private/otel.key
          min_version: "1.3"  # Enforce TLS 1.3

Authentication Setup

extensions:
  basicauth/server:
    htpasswd:
      file: /etc/otel/auth.htpasswd

receivers:
  otlp:
    protocols:
      grpc:
        auth:
          authenticator: basicauth/server

Deployment Considerations

Kubernetes Gotchas

• Operator Issues: Silent failures with no error messages
• Config Validation: Use --dry-run to catch YAML errors
• Resource Limits: Set memory limits to prevent node crashes

Docker Deployment

# Proper resource limits
docker run --memory=2g --cpus=1 otel-collector

# Persistent storage for queues
docker run -v /var/lib/otel-data:/var/lib/otel-data otel-collector

Real-World Lessons

Black Friday Incident

• Problem: Deployed without persistent queues
• Impact: 4 hours of telemetry data lost during backend outage
• Solution: Always enable file_storage for production deployments

Memory Explosion Case

• Problem: Developer added user IDs to metrics
• Impact: Prometheus storage: 100GB → 2TB overnight, queries unusable
• Prevention: Implement high-cardinality filtering from day one

Version Upgrade Failure

• Problem: Upgraded to v0.89.0 during high-traffic period
• Impact: Memory leaks caused collector crashes every 2 hours
• Prevention: Never use .0 releases, always test in staging first

Success Metrics

Cost Reduction Achieved

• Previous: $18k/month Datadog bill
• Current: ~$5k/month mixed backends (73% reduction)
• Method: 90% data to Grafana Cloud, 10% critical data to Datadog

Operational Benefits

• Vendor switching: 2 hours config change vs months of code rewriting
• Data control: PII filtering, sampling, enrichment before export
• Reliability: Built-in retries, queuing, no data loss during outages

This technical reference preserves all operational intelligence while organizing it for AI consumption, including specific failure modes, resource requirements, and real-world implementation lessons.