OpenLIT - Monitor Your LLMs and GPUs

OpenLIT monitors your AI apps without the usual observability hell. Been running it for 8 months - here's what actually matters.

The Problem It Solves

Your LLM costs are spiraling out of control and you have no idea why. That GPT-4 call that should cost $0.03 is somehow costing $3.00 because someone's feeding it a 50-page PDF and the retry logic is completely fucked. Your GPU training job crashed at 90% completion and you don't know if it was OOM, driver issues, or thermal throttling.

OpenLIT catches this stuff before it costs you money or sleep. The observability gap in AI systems is a real problem - traditional APM tools weren't built for token-based pricing models or GPU memory profiling.

Zero-Code Setup (Actually Works)

Most "zero-code" observability is bullshit. OpenLIT's actually works:

## Instead of: python app.py  
openlit-instrument python app.py

That's it. No SDK imports, no configuration files, no wrestling with OpenTelemetry collectors. It auto-detects 50+ integrations including OpenAI, Anthropic, LangChain, ChromaDB, and whatever vector database you're using this week.

The magic is it hooks into HTTP requests and catches API calls automatically. Works 90% of the time - the other 10% you're debugging OTLP endpoints, but that beats manual instrumentation. The OpenTelemetry semantic conventions for AI workloads are still evolving, but OpenLIT handles the complexity for you. Unlike traditional tracing approaches, you don't need to instrument every LangChain call manually.

Cost Tracking That Doesn't Lie

OpenLIT pulls actual token counts from API responses instead of estimating. Saved us from a $5k OpenAI bill when we discovered a retry loop was sending the same massive context 400 times.

Custom pricing works too - we track our fine-tuned models with accurate per-token costs. Cost calculations lag 5-10 seconds on large datasets but that's acceptable for budget monitoring. The cost optimization capabilities beat most dedicated FinOps tools. Unlike basic monitoring solutions, you get granular cost breakdowns per user session, model, and request type. The pricing documentation shows how to configure custom model costs, while OpenTelemetry cost monitoring patterns explain implementation details. For enterprise cost tracking, the Grafana Cloud integration provides advanced analytics.

GPU Monitoring for Local Models

If you're running local models, GPU monitoring is essential. OpenLIT tracks NVIDIA and AMD GPUs - utilization, memory, temperature, power draw. Requires driver 470.x+ on NVIDIA, older drivers will randomly stop reporting metrics.

Caught a runaway training job that was thermal throttling at 83°C. Would've taken 3x longer without monitoring. The GPU observability integration gives you the same depth as dedicated tools like nvidia-ml-py, but correlates with your LLM traces. Better than separate monitoring approaches that don't connect GPU metrics to specific inference requests. The GPU monitoring documentation covers setup details, while NVIDIA GPU observability patterns show integration approaches. For production GPU deployments, check the Kubernetes GPU monitoring guide and Docker GPU setup documentation.

The Gotchas

Port 4318 conflicts with other OTLP collectors - plan for that. ClickHouse eats RAM like crazy, budget 32GB for production or it'll OOM during trace aggregations.

Dashboard gets slow with >1M traces, use time filters. Network latency to OTLP endpoint kills performance if you're sending traces across continents.

Docker Setup (Works Until It Doesn't)

Docker Compose works great for dev environments:

git clone https://github.com/openlit/openlit.git
cd openlit
docker compose up -d

Takes 2 minutes on my M1 Mac, 20 minutes on the company's ancient Intel box if ClickHouse decides to be a pain. Default login is user@openlit.io / openlituser.

The ClickHouse container can get stuck in error state during startup. Just wait - it's usually the database taking forever to initialize. If it's still broken after 10 minutes, check if you have enough disk space. ClickHouse is picky about storage. The official deployment docs cover most edge cases, but the Docker troubleshooting guide has the real solutions. For production Docker setups, follow the Docker Compose best practices and review the ClickHouse Docker optimization guide. The OpenTelemetry Collector Docker deployment explains OTLP endpoint configuration, while the observability stack deployment patterns cover integration approaches.

Kubernetes (When You Hate Yourself)

Helm chart exists but comes with the usual k8s gotchas:

helm repo add openlit https://artifacthub.io/packages/helm/openlit
helm install openlit openlit/openlit

Memory requirements are brutal - ClickHouse needs 8GB minimum or it'll OOM during aggregations. The operator can auto-inject instrumentation but breaks when pods don't have proper RBAC permissions. The Kubernetes setup guide covers most deployment scenarios, and the Helm values configuration lets you tune resource limits properly.

Configuration Hell

The zero-code approach works 90% of the time:

openlit-instrument python app.py

When it doesn't work, you're debugging OTLP collector endpoints. Port 4318 conflicts with everything - Jaeger, other collectors, your local dev proxy. Pick a different port and configure it. The OpenTelemetry troubleshooting docs are essential reading, and the collector configuration examples cover most common setups:

import openlit

openlit.init(
    otlp_endpoint=\"http://localhost:4320\",  # Not 4318
    environment=\"production\"
)

Performance Impact (The Truth)

"Less than 5ms latency" is marketing bullshit. In reality:

• Local LLM calls: ~2-5ms overhead
• Remote API calls: Negligible (API latency dominates)
• High-throughput apps: Can add 10-20ms during trace ingestion spikes
• Memory usage: +50-100MB per process

What Breaks in Production

ClickHouse Memory Issues: Plan for 32GB RAM minimum. We crashed production twice before learning this. Trace ingestion can spike memory usage 5x during burst periods. The ClickHouse performance tuning guide has the settings that actually matter.

Network Latency: OTLP endpoint across regions kills performance. Keep collectors geographically close to your apps. The distributed tracing patterns explain why latency compounds in AI workloads.

GPU Monitoring Fails: Randomly stops working after NVIDIA driver updates. Requires container restart. AMD GPU support is newer and breaks more often. The GPU monitoring documentation covers driver compatibility matrices.

Storage Growth: 10GB per million traces quickly becomes terabytes. Set up log rotation or your disk will fill up. This killed our staging environment - learned that lesson the hard way. The storage optimization guide covers retention policies that actually work.

The 3AM Debugging Checklist

When OpenLIT stops working (not if):

1. Check if port 4318 is actually listening: netstat -tulpn | grep 4318
2. ClickHouse out of memory? Check container logs
3. OTLP collector reachable? curl http://localhost:4318/v1/traces
4. GPU monitoring dead? Restart containers, check driver version

Time estimate: 15 minutes if you know what you're doing, 2 hours if you don't.