Arize AI: ML & LLM Production Monitoring - Technical Reference

What Arize Does

Production monitoring for ML models and LLMs that detects failures before user complaints. Tracks data drift, performance degradation, and infrastructure issues across traditional ML and LLM applications.

Deployment Options

Phoenix (Open Source)

• Cost: Free + infrastructure hosting costs
• Setup Time: 10 minutes if successful, 2 hours with common issues
• Common Issues:
  • ModuleNotFoundError with opentelemetry in wrong virtual environment
  • Docker networking conflicts
  • Port conflicts with TensorBoard (both default to localhost:6006)
• Performance Impact: 10-50ms latency overhead, 5-10MB memory per process
• Data Limits: Unlimited (self-hosted storage)
• Best For: Prototyping, small teams, infrastructure control preference

Arize AX (Hosted)

• AX Free: 25k spans, 1 week retention, single user
• AX Pro: $50/month, 100k spans, 2 weeks retention, 3 users max
• AX Enterprise: $1000+/month, unlimited data, enterprise compliance

Critical Failure Modes

LLM-Specific Failures

1. Prompt Version Regression: V2 prompts break working V1 functionality
2. Token Cost Explosion: Recursive loops can burn $1,100+ over weekends
3. Agent Infinite Loops: get_weather → analyze_weather → get_weather cycles hit Lambda timeouts
4. Hallucination at Scale: Models confidently provide dangerous advice (medical, legal)

Traditional ML Failures

1. Data Drift: Input distributions change, model accuracy drops to 60%
2. Feature Engineering Bugs: age_in_years becomes age_in_days, model thinks 25-year-olds are 9,125 years old
3. Embedding Collapse: All recommendations cluster to single category
4. Silent Bias Creep: Models develop discriminatory patterns over time

Infrastructure Failures

1. Memory Pressure: Feature extraction timeouts return zeros, causing garbage predictions
2. Instance Type Changes: Switching to expensive GPU instances can increase costs from $200 to $2000+/month
3. High-Frequency Impact: >1000 RPS systems may see 95th percentile latency increase from 180ms to 230ms

Production Implementation Requirements

Setup Prerequisites

• OpenTelemetry support in existing framework
• Manual tracing for custom frameworks
• API key management for hosted version
• Instrumentation code additions (typically 3 lines)

Performance Thresholds

• Acceptable Latency Impact: 10-50ms for LLM applications (2-5 second baseline)
• Memory Overhead: 5-10MB per process
• Critical Threshold: Test impact before implementing on >1000 RPS systems
• Emergency Disable: OTEL_SDK_DISABLED=true stops tracing without deployment

Alert Configuration

• Useful Alerts: accuracy below 70%, cost per request spikes
• Avoid: micro-fluctuations (0.1% accuracy changes)
• Critical Metrics: confidence distribution changes, token usage patterns, embedding drift

Framework Compatibility

Well-Supported

• OpenAI, Anthropic, major cloud providers
• LangChain, LlamaIndex (good integration)
• Frameworks with existing OpenTelemetry support

Limited Support

• CrewAI and newer frameworks (integration bugs expected)
• Custom in-house frameworks (manual tracing required)
• Legacy systems without OTEL (significant development overhead)

Cost Analysis

Hidden Costs

• Infrastructure hosting for Phoenix
• Development time for custom framework integration
• Alert fatigue from misconfigured thresholds
• Compliance overhead for enterprise features

ROI Scenarios

• Prevented Customer Churn: Early detection of recommendation system failures
• Cost Control: Token usage monitoring prevents runaway API charges
• Debugging Efficiency: Trace visualization reduces debugging from hours to minutes
• Compliance Value: Bias detection prevents discriminatory model behavior

Risk Mitigation

Data Security

• Traces contain model inputs/outputs (avoid PII)
• Self-hosted Phoenix for sensitive data
• SOC2/HIPAA compliance available in Enterprise tier
• Review data processing agreements for regulated industries

Operational Risks

• Service Dependency: AX outages eliminate monitoring visibility
• Vendor Lock-in: Trace format migration complexity
• False Negatives: Auto-instrumentation works ~80% of time
• Scale Limitations: Free tier 25k spans exhausted quickly in production

Decision Matrix

Use Case	Recommendation	Reasoning
Prototype/Development	Phoenix OSS	Free, full features, learning curve acceptable
Small Production Team	AX Pro ($50/month)	Managed infrastructure, team collaboration
Enterprise Compliance	AX Enterprise	Required certifications, unlimited scale
High-Frequency ML	Evaluate Impact First	Latency sensitivity requires testing
Sensitive Data	Phoenix Self-Hosted	Data sovereignty requirements

Critical Success Factors

Implementation

1. Start with basic tracing before advanced features
2. Configure conservative alert thresholds initially
3. Test performance impact in staging environment
4. Plan manual tracing for unsupported frameworks

Operational

1. Monitor token costs from day one
2. Set up embedding drift detection early
3. Implement bias monitoring for user-facing models
4. Document prompt versions for rollback capability

Scaling

1. Evaluate retention needs before choosing tier
2. Plan for enterprise compliance requirements
3. Consider multi-region deployment for critical systems
4. Budget for data volume charges in enterprise pricing