Datadog Production Troubleshooting - When Everything Goes to Shit

When Your Agents Stop Talking to Datadog (And Why)

The most brutal production failures happen when your monitoring goes dark exactly when you need it most. Your agents were sending data fine for months, then suddenly nothing. Your dashboards show flat lines while your infrastructure is melting down.

Here's what actually goes wrong and how to fix it before you get paged at 3am:

The "Agent Is Running But No Metrics" Nightmare

The Silent Agent Problem: Your monitoring dashboards show flat lines across all metrics - CPU, memory, network, everything at zero. But sudo systemctl status datadog-agent reports the service is running normally with no error messages. The process is alive but completely braindead.

This one is fucking evil because sudo systemctl status datadog-agent says everything's fine, but your dashboards are empty. The agent process is alive but braindead.

Check the agent status first: sudo datadog-agent status will tell you what's actually happening. Look for this output:

• Forwarder: Should say "Running" with recent transactions
• Collector: Should list active checks
• DogStatsD: Should show recent metrics received

If the forwarder is failing, you've got network issues. If collector shows no checks, your agent config is fucked.

Common causes:

• Clock skew: Datadog rejects metrics with timestamps >10 minutes off. Run ntpdate -s time.nist.gov and restart the agent
• API key rotation: Someone rotated keys but forgot to update agents. Check /etc/datadog-agent/datadog.yaml for the correct api_key
• Firewall changes: Your network team blocked egress to https://app.datadoghq.com without telling anyone. Use curl -v https://app.datadoghq.com to test connectivity

The official troubleshooting guide covers the basics, but here's what they don't tell you:

Memory pressure kills agents: If your host is swapping, the agent will start dropping metrics silently. Monitor agent memory usage with ps aux | grep datadog-agent - if resident memory is >500MB, something's wrong.

Log rotation breaks everything: When logrotate runs, it can kill agent log handlers. Check /var/log/datadog/agent.log for sudden stops in logging. You'll see "ERROR: log file descriptor closed unexpectedly" followed by silence. Restart the agent after log rotation, or better yet, configure logrotate to send SIGHUP to the agent process instead of just rotating files.

High Memory Usage: When Your Agent Becomes a Memory Hog

Datadog agents are supposed to be lightweight, but in production they can eat gigabytes of memory and nobody knows why. I've seen agents consume 8GB RAM on a 16GB host, essentially DOSing the applications they're monitoring.

Quick diagnosis: Run sudo datadog-agent status and look for:

• Forwarder queue size: Should be <1000. Higher means backlog
• Check collection time: Individual checks taking >30s indicate problems
• DogStatsD buffer: High buffer usage means metric flood

The high memory troubleshooting docs are actually useful here, but miss the real production culprits.

APM trace explosion: The biggest memory killer is applications sending massive traces. One shitty microservice generating 10,000-span traces will crash your agent. Check APM resource usage and configure trace sampling immediately.

Custom metrics cardinality bomb: When someone instruments metrics with user_id tags, you get millions of unique timeseries. Each costs memory. Use datadog-agent check system_core to see how many metrics you're generating.

Log tailing gone wrong: If you're tailing massive log files, the agent buffers everything in memory. Configure log processing rules to filter at the agent level, not after ingestion.

Fix it before it kills your host:

1. Set agent memory limits in systemd: MemoryMax=2G in /etc/systemd/system/datadog-agent.service.d/memory.conf
2. Configure DogStatsD buffer size to prevent metric floods
3. Enable agent resource limits in the config

Kubernetes Agent Clusterfuck

Kubernetes Complexity Layers: The Datadog agent deploys as a DaemonSet (one per node) plus a cluster agent for metadata aggregation, all coordinating through the Kubernetes API server. When any layer fails, the whole monitoring stack goes dark, usually during your worst production incidents.

Kubernetes adds layers of complexity that break in spectacular ways. The cluster agent sounds great until it decides to crash during your biggest production incident.

DaemonSet deployment problems: Despite Datadog's warnings, many teams still use manual DaemonSet deployment because "it's simpler." It's not. It breaks in ways that waste weeks.

Common DaemonSet failures:

• RBAC permissions: Agent pods fail to start with vague "forbidden" errors. Check cluster role bindings
• Node selector conflicts: Agents don't deploy to new nodes because selector rules exclude them
• Resource limits: Kubernetes kills agent pods under memory pressure if limits are too low

Use the fucking Operator: The Datadog Operator exists for a reason. It handles RBAC, node selectors, and resource management automatically.

Cluster Agent crashes: The cluster agent is supposed to aggregate metadata and reduce API load, but it crashes when overwhelmed. Check cluster agent logs for:

• too many open files: Increase file descriptor limits
• context deadline exceeded: Kubernetes API is slow, increase timeouts
• authentication failed: Service account tokens expired

Container discovery problems: Agents discover containers automatically, but this breaks with:

• Custom networking: CNI plugins that hide containers from discovery - Calico with strict network policies makes agents return empty container lists
• Namespace restrictions: RBAC that prevents cross-namespace discovery - agents throw "forbidden: pods is forbidden" errors that don't show in agent status
• Container runtime changes: Docker to containerd migrations break agent configs - agent 7.41.0 has a known issue where containerd socket detection fails on Ubuntu 22.04 with non-standard socket paths

Fix by enabling container troubleshooting and checking agent autodiscovery rules.

The Proxy Hell

Corporate networks require proxy servers, and Datadog agents hate proxies with the fury of a thousand suns. The proxy configuration docs make it sound easy. It's not.

SSL interception breaks everything: Your corporate proxy intercepts SSL and presents its own certificates. Datadog agents reject these and fail silently.

Fix with proxy bypass rules or certificate pinning workarounds:

## /etc/datadog-agent/datadog.yaml
skip_ssl_validation: true  # Only for desperate situations
proxy:
  https: "http://proxy.company.com:8080"
  http: "http://proxy.company.com:8080"

Proxy authentication: NTLM authentication makes agents sad. Use basic auth if possible, or configure proxy bypass for Datadog endpoints.

Split tunneling disasters: When VPN routes some traffic through proxy and some direct, agents get confused about which path to use. You'll see intermittent "connection refused" errors in agent logs but everything looks configured correctly. Create explicit routing rules for Datadog IPs, or better yet, avoid split tunnel VPNs entirely if you want reliable monitoring.

Performance That Destroys Production

Monitoring shouldn't make your production systems slower, but badly configured Datadog can destroy performance in subtle ways.

Check interval abuse: Running checks every 10 seconds sounds reasonable until you have 50 checks per host. Database connection checks that run too frequently can exhaust connection pools.

Configure sensible intervals:

instances:
  - host: localhost
    port: 5432
    min_collection_interval: 60  # seconds, not 10

Log tailing performance: Tailing high-volume log files kills disk I/O. Use log sampling aggressively and filter garbage logs at the agent level.

Network saturation: Agents sending metrics to Datadog consume bandwidth. In bandwidth-constrained environments, configure compression and adjust flush intervals to batch network calls.

The key insight: Datadog works great until it doesn't, and when it fails, it fails in production during incidents when you need it most. Build redundancy and monitoring for your monitoring. Keep a simple external check that verifies your agents are actually sending data, not just running processes.

Next, let's look at the dashboard and UI performance problems that make troubleshooting even harder when you're already having a bad day.

When Your Monitoring Bill Becomes Breaking News

The worst production disasters aren't infrastructure failures - they're surprise Datadog bills that make your CFO schedule emergency meetings. I've watched teams get $200k annual renewals when they expected $20k. Here's how costs spiral out of control and how to fix it before you get fired.

Custom Metrics: The Silent Budget Killer

Custom metrics are marketed as "just $5 per 100 metrics" but that's bullshit in production. Each unique tag combination creates a separate billable metric. Tag a counter with user_id and suddenly you're paying for every user in your database.

The cardinality explosion scenario: A team instruments a login counter with tags for user_id, device_type, browser, and region. Seems reasonable until you realize:

• 100,000 users × 5 device types × 10 browsers × 20 regions = 100 million metrics
• At $0.05 per metric monthly, that's $5 million annually for one fucking counter

Cardinality Math Explosion: Each unique combination of tags creates a separate billable metric. The mathematical explosion happens when you multiply tag values together - what starts as "just a few tags" becomes millions of unique combinations. It's a combinatorial explosion that destroys budgets overnight.

How to detect the problem before it destroys your budget: Use metrics without limits to identify high-cardinality metrics. Check your billing dashboard weekly, not when renewal comes up.

Strategic tagging that saves money: Replace high-cardinality tags with business-relevant groupings:

• Replace user_id:12345 with user_tier:premium
• Replace container_id:abc123 with service:user-api
• Replace request_id:uuid with endpoint:/api/users

You get the same business insights at 1% of the cost.

Governance that prevents disasters: Require approval for custom metrics after 1,000 per application. Create automated alerts when monthly custom metrics grow >20%. Most importantly, make teams own their metric costs through chargeback.

APM Span Costs: When Tracing Bankrupts You

APM spans cost $0.0012 each, which sounds cheap until you realize microservices generate millions of spans daily. One badly configured service can cost $100k annually in span ingestion alone.

The microservice multiplier effect: Each microservice call creates multiple spans:

• Incoming HTTP request span
• Database query spans (one per query)
• Outgoing HTTP spans to other services
• Cache operation spans
• Background job spans

A simple user login might generate 50+ spans across 8 microservices. At 1 million logins monthly, that's 50 million spans = $60k annually for one user flow. Learned this when our user signup flow started costing more to monitor than our entire compute bill - turns out the email verification service was generating 200 spans per signup because someone instrumented every database query and Redis call.

Smart sampling saves budgets: Use trace sampling aggressively:

• 100% sampling for errors and slow requests (>2s)
• 10% sampling for normal requests
• 1% sampling for health checks and monitoring endpoints

Configure service-level sampling based on business importance. Critical user-facing services get higher sampling than internal background jobs.

Span filtering at the agent: Configure the agent trace pipeline to drop useless spans before they hit Datadog's billing system:

apm_config:
  filter_tags:
    reject:
      - "http.url:/health"  # Drop health check traces
      - "resource.name:GET /ping"  # Drop ping endpoints

Log Costs: The Data Volume Apocalypse

The Log Volume Death Spiral: Applications start logging conservatively but gradually get more verbose as developers add debug statements. Log volume follows an exponential growth pattern - starts manageable, then suddenly explodes as microservices multiply and debug logging accumulates across services.

Log costs explode faster than any other Datadog service because applications are chatty as fuck. At $1.27 per million events, debug logging from microservices becomes a six-figure annual expense.

Debug logging will bankrupt you: A single Node.js application with debug logging enabled can generate 10 million log events daily:

• 10M events/day × 365 days = 3.65 billion events annually
• 3.65B events × $1.27 per million = $4,635 annual cost per application
• With 50 microservices, that's $231k annually for debug logs nobody reads

Sampling and filtering saves your job: Use log processing rules to filter at the agent level:

• Sample INFO logs at 10% (keep all ERROR/WARN)
• Sample DEBUG logs at 1% (or 0% in production)
• Exclude health checks, successful authentication, and scheduled job success logs entirely

Strategic log levels prevent disasters:

• ERROR: Always keep 100% - you need these during incidents
• WARN: Keep 100% - indicates potential problems
• INFO: Sample at 10% - gives you operational insight without flooding
• DEBUG: Sample at 1% maximum - only for active debugging scenarios

The New Flex Logs Strategy

Datadog's Flex Logs tiers help control long-term costs but require planning:

Hot Tier (0-15 days): Full search and real-time alerts. Budget 60% of log costs here for operational data, errors, and security events.

Frozen Tier (15+ days): Searchable but slower. Perfect for compliance retention and post-incident analysis. Can reduce storage costs 70% while meeting audit requirements.

Archive Tier (90+ days): S3/GCS storage with Datadog metadata. Cheapest option for long-term compliance but requires rehydration for complex queries.

Configure retention policies before deploying, not after ingesting terabytes of expensive data.

Resource Optimization: Making Agents Behave

Datadog agents can become resource hogs that impact application performance. Proper resource management prevents monitoring from slowing down production systems.

Memory management prevents disasters: Agents buffer metrics, logs, and traces in memory. Without limits, they'll consume all available RAM:

## /etc/datadog-agent/datadog.yaml
forwarder_timeout: 20
forwarder_retry_queue_max_size: 100
log_file_max_size: 10MB
log_file_max_rolls: 5

Set systemd memory limits to prevent runaway agents:

## /etc/systemd/system/datadog-agent.service.d/memory.conf
[Service]
MemoryMax=2G
MemoryHigh=1.5G

CPU optimization for production loads: Agents doing too much work slow down applications sharing the same host:

• Increase check intervals for non-critical metrics (60s instead of 10s)
• Disable unused integrations that generate metrics you don't need
• Use DogStatsD buffer configuration to batch network calls

Network bandwidth considerations: In bandwidth-constrained environments, configure compression and batching:

compression: gzip
forwarder_retry_queue_max_size: 1000
batch_max_concurrent_send: 10

Multi-Cloud Cost Optimization

Enterprise deployments across AWS, Azure, and GCP face additional cost multipliers from data transfer charges.

Regional data locality: Deploy agents in the same regions as workloads to minimize cross-region transfer costs. One misconfigured agent cluster generating cross-region traffic can cost $10k+ monthly in cloud provider egress charges.

Cloud-native integration strategy: Use cloud provider integrations (CloudWatch, Azure Monitor) for basic infrastructure metrics instead of agent-based collection where possible. This reduces compute costs and agent maintenance overhead.

Proxy infrastructure optimization: For organizations requiring proxy deployment, size proxy resources properly to handle peak telemetry loads without becoming bottlenecks during incidents.

Budget Planning and Predictive Cost Control

Growth modeling that accounts for reality: Infrastructure monitoring costs scale predictably, but application monitoring explodes non-linearly:

• 20% customer growth = 50% more containers due to auto-scaling
• One new microservice = 10x more spans due to service mesh overhead
• Feature launches = 5x more custom metrics from A/B testing

Automated cost controls prevent disasters: Configure usage limits that automatically increase sampling when approaching budget thresholds:

• Emergency sampling: 50% reduction in trace ingestion when costs hit 80% of budget
• Log filtering: Automatic exclusion of non-critical logs at 90% of budget
• Metric freeze: Stop accepting new custom metrics at 95% of budget

Team accountability through chargeback: Use usage attribution to allocate costs to teams based on tags. When teams see their actual monitoring costs, behavior changes quickly.

Create cost dashboards that show:

• Daily spending rate vs monthly budget
• Top cost drivers by team and application
• Growth trends for each billable service

The key insight: Datadog costs are primarily driven by data volume and cardinality, not infrastructure size. You can monitor 1,000 hosts cheaply or bankrupt your company monitoring 10 hosts that generate high-cardinality metrics and verbose logs.

Focus optimization efforts on the applications generating the most billable events, not the infrastructure running them. The biggest cost savings come from developers changing how they instrument code, not ops teams tuning agent configurations.

Understanding what drives costs helps you optimize before renewal negotiations. Speaking of which, let's look at the operational optimization techniques that keep Datadog performing well under production loads.