LLM Production Monitoring: AI-Optimized Technical Guide

Executive Summary

Traditional APM tools fail for LLM monitoring because they cannot detect semantic failures, cost spirals, or quality degradation. Production LLM monitoring requires specialized tooling and costs $500-6000/month for reliable implementation.

Critical Failure Modes

Silent Failures That Break Production

• Rate Limiting Without Error Codes: OpenAI throttles requests to crawl speed without returning error codes, showing HTTP 200 while users wait 30+ seconds
• Token Usage Explosions: Single user queries can trigger 10,000-token responses, increasing daily costs from $200 to $2,800 overnight
• Prompt Injection Attacks: Users bypass instructions to generate expensive long-form content, burning $3,000+ before detection
• Model Update Breaking Changes: Provider model updates change response formats without warning, breaking parsing logic in production
• Empty Response Failures: Silent rate limit hits cause 30% blank responses for 6+ hours while traditional metrics show 99.9% uptime

Cost Spiral Indicators

• Alert Threshold: 150% of daily normal spend (not 500% - too late for recovery)
• Token Usage Spikes: Average request jumping from 500 to 5,000+ tokens indicates system abuse
• Weekend Debugging: Forgotten test scripts can generate $1,200+ in charges during off-hours

Technical Implementation Stack

Required Infrastructure

• Kubernetes: v1.24+ minimum (v1.23 breaks compatibility)
• Memory: 16GB minimum (8GB insufficient for Prometheus memory consumption)
• Storage: 200GB+ for metrics retention (retention policies are unreliable)
• Python: 3.9+ required (3.8 breaks OpenTelemetry in production)
• Node.js: 18+ required (LLM libraries incompatible with v16)

Core Monitoring Components

OpenTelemetry Collector (Primary Failure Point)

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  memory_limiter:
    limit_mib: 512  # Increase to 2048 if OOMing
  batch:
    send_batch_size: 1024  # Reduce if "batch too large" errors
service:
  pipelines:
    metrics:
      processors: [memory_limiter, batch]  # Order critical

Common Failures:

• connection refused on port 4317: Docker network configuration failure
• jaeger-collector not found: DNS resolution failed, use IP addresses
• memory limit exceeded: Double memory_limiter value
• batch processor failed: Reduce batch size for oversized traces

Prometheus Configuration

• Retention: Maximum 7 days (consumes all disk space beyond this)
• Scrape Interval: 15 seconds minimum (shorter intervals crash Prometheus)
• Sampling Rate: 1% maximum for traces (100% fills storage within days)

Jaeger Tracing

• Development: Works reliably
• Production: Frequent "trace collection timed out" errors
• Storage: Default sampling rate (100%) requires immediate reduction to 1%

LLM-Specific Monitoring Tools

OpenLIT (Recommended)

• Advantage: Auto-instrumentation without manual API call wrapping
• Compatibility: OpenAI, Anthropic, LangChain, most major providers
• Implementation: 2-line code addition for full instrumentation

Alternative Solutions by Budget

Solution Type	Monthly Cost	Setup Time	Reliability
DIY Everything	$300-800	2-3 days	Frequent failures
Hybrid Approach	$800-2200	15-30 hours setup	Weekend debugging required
Full Commercial	$2000-6000	5-15 hours	Vendor lock-in risk

Provider-Specific Support Quality

• OpenAI: Universal support across all tools
• Anthropic: Universal support across all tools
• Google Cloud AI: Inconsistent, half the metrics often broken
• AWS Bedrock: Usually broken or incomplete implementation
• Azure OpenAI: Mostly functional (Microsoft-branded OpenAI)
• Local Models (Ollama/vLLM): Minimal support, requires custom instrumentation

Critical Monitoring Metrics

Immediate Alert Thresholds

• Error Rate: >5% (not 1% - reduces false positives)
• Response Time: >30 seconds (users abandon after 15 seconds)
• Cost Spike: >200% of daily average (not 20% - normal variance)
• Service Availability: No successful requests in 5 minutes

Secondary Alert Thresholds

• Token Usage: 3x increase per request (indicates prompt gaming)
• GPU Memory: >90% for local models
• Disk Space: <10GB remaining (Prometheus storage consumption)

Security Requirements

Data Protection Implementation

• Sensitive Data Regex Patterns:
  • SSN: \b\d{3}-\d{2}-\d{4}\b
  • Credit Cards: \b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b
  • Email: \b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b

Compliance Requirements

• Data Minimization: Hash sensitive fields instead of exclusion for debugging capability
• Audit Logging: Permanent storage of access logs (user ID, timestamp, data accessed)
• Access Control: OAuth2/SAML for enterprise, never admin/admin defaults
• Encryption: All data in transit, VPN-protected monitoring dashboards

Cost Control Implementation

Real-Time Cost Prevention

# Rate limiting by estimated cost per request
rate_limits = {
    "gpt-4": {"max_requests": 10, "window": "1h"},
    "gpt-3.5-turbo": {"max_requests": 100, "window": "1h"}
}

Model Selection Rules

• <50 words: Route to cheap models
• >500 words: Route to expensive models
• Accuracy: 70% correct routing (30% error rate acceptable for 50% cost reduction)

Dashboard Configuration

Executive Dashboard

• Primary Metric: Daily cost
• Secondary Metric: System uptime status
• Design Rule: Single number focus, no technical details

Operations Dashboard (3AM Debugging)

• API Error Rates: Visual threshold at 1%
• Response Time Percentiles: Never use averages (misleading)
• Token Usage: Per-minute monitoring
• Alert Status: Large, red, unmistakable failure indicators

Engineering Dashboard

• Request Traces: Individual transaction timing breakdowns
• Token Usage: Per-request granularity
• Quality Scores: Model response evaluation metrics
• Edge Cases: Log unusual user interaction patterns

Troubleshooting Checklist (3AM Debugging)

Primary System Checks

1. Docker Status: docker ps - daemon frequently fails silently
2. Port Accessibility: telnet localhost 4317 and telnet localhost 4318
3. Data Flow: Check OTel Collector logs for "received spans" messages
4. Prometheus Targets: Visit http://localhost:9090/targets for scraping status
5. Disk Space: df -h - Prometheus storage consumption

Error Message Decoder

• "connection refused": Service down or firewall blocking
• "no route to host": DNS failure, switch to IP addresses
• "context deadline exceeded": Resource exhaustion, check limits
• "batch processor failed": Trace size too large, reduce batch size
• "memory limit exceeded": OTel Collector OOM, increase RAM allocation

Resource Requirements

Minimum Production Setup

• Budget: $500/month minimum for functional monitoring
• Personnel: 2-3 days initial setup time
• Maintenance: 15-30 hours monthly for hybrid approach
• Hidden Costs: AWS storage doubling, weekend debugging time

Tool-Specific Costs

• Datadog Custom Metrics: $5/month per metric (50+ LLM metrics typical)
• Grafana Cloud: $200-500/month metric ingestion
• Third-party LLM Monitoring: $500-2000/month additional

Quality Monitoring

Automated Quality Checks

• LLM-on-LLM Evaluation: GPT-3.5-turbo evaluating GPT-4 responses ($0.01/evaluation)
• Response Length Monitoring: Track average response length changes
• Refusal Rate Tracking: Monitor responses containing "I cannot" or "I'm not able to"
• Repetition Detection: Alert when boilerplate appears in >20% of responses

Local Model Monitoring

• vLLM: Prometheus metrics available with --metric-port 8000
• Ollama: No built-in monitoring, system metrics only
• GPU Memory: Critical monitoring for OOM prevention
• Queue Depth: Alert at >10 requests (response time degradation)
• Model Loading: 30+ second load times appear broken to users