Prometheus Node Exporter Production Configuration Guide

Critical Configuration Requirements

Default Configuration Failures

• Default Risk: Node Exporter v1.9.1 ships with 70+ collectors enabled by default
• Production Impact: Causes 8GB+ RAM usage on 200-node clusters, can crash Prometheus servers
• Memory Growth Pattern: interrupts collector generates 500+ metrics per server, slabinfo can consume 2GB+ memory alone
• Kernel Stability: Parallel I/O operations crash Linux kernels on 96-core servers (documented in GitHub issue #2530)

Essential Collector Configuration

Production-Safe Minimal Setup:

./node_exporter \
  --collector.disable-defaults \
  --collector.cpu \
  --collector.meminfo \
  --collector.filesystem \
  --collector.diskstats \
  --collector.netdev \
  --collector.loadavg

Critical Filesystem Filtering (Required for Docker/Kubernetes):

--collector.filesystem.mount-points-exclude="^/(sys|proc|dev|host|etc|var/lib/docker)($$|/)"
--collector.filesystem.fs-types-exclude="^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"

Collector Decision Matrix

Collector	Memory Usage	Production Value	Critical Warnings
cpu	10MB	Essential	Basic alerting foundation
meminfo	5MB	Essential	Memory leak detection
filesystem	200MB+ without filtering	Critical	Prevents disk full disasters; MUST filter Docker mounts
diskstats	50MB	High	Detects database I/O thrashing
netdev	100MB	High	Bandwidth saturation alerts; filter Docker interfaces
pressure	20MB	High	Modern load indicators (PSI stall metrics)
hwmon	50MB	Medium	Temperature/fan monitoring; filter voltage readings
interrupts	500MB+	Rarely useful	Crashes on 96-core systems
slabinfo	1GB+	Never useful	Kernel debugging only
ethtool	200MB	Rarely useful	Can cause packet drops
systemd	100MB	Medium	Service status monitoring on systemd systems

Performance Thresholds and Failure Points

Critical Performance Metrics

• Maximum metrics per node: 2000 (above this causes Prometheus performance degradation)
• Memory limit per instance: 200MB (production deployments should set container limits)
• Scrape timeout threshold: 5 seconds (longer indicates too many collectors enabled)
• Cardinality explosion pattern: 200 nodes × 2000 metrics = 400k series (Prometheus failure point)

Single-Threading Constraint

• GOMAXPROCS=1 enforced since v1.5.0: Parallel I/O operations crash Linux kernels
• Performance trade-off: Slower scraping prevents kernel panics on high-core systems
• Cannot override: Attempting to increase GOMAXPROCS causes system instability

Kubernetes Deployment Requirements

Essential Host Access Configuration

spec:
  hostNetwork: true
  hostPID: true
  securityContext:
    runAsUser: 65534  # nobody user
    runAsNonRoot: true
  containers:
  - name: node-exporter
    args:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/host/root'
      - '--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)'
    volumeMounts:
    - name: root
      mountPath: /host/root
      mountPropagation: HostToContainer  # Critical: missing causes mount detection failures
      readOnly: true

Resource Limits (Required)

resources:
  limits:
    memory: 200Mi
  requests:
    cpu: 100m
    memory: 100Mi

Security Implementation Requirements

Network Security (Critical)

# DON'T bind to all interfaces (security vulnerability)
./node_exporter --web.listen-address="192.168.1.100:9100"

# TLS configuration (mandatory for production)
./node_exporter \
  --web.config.file=/etc/node_exporter/web.yml \
  --web.listen-address=":9100"

TLS Configuration Template

tls_server_config:
  cert_file: /etc/ssl/certs/node_exporter.crt
  key_file: /etc/ssl/private/node_exporter.key
  min_version: TLS12
  cipher_suites:
    - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384

Systemd Security Hardening

[Service]
NoNewPrivileges=true
PrivateTmp=true
ProtectHome=true
ProtectSystem=strict
ReadWritePaths=/var/lib/node_exporter
CapabilityBoundingSet=CAP_DAC_OVERRIDE
AmbientCapabilities=CAP_DAC_OVERRIDE
LimitNOFILE=8192
MemoryLimit=512M

Textfile Collector Implementation

Atomic File Writing (Required)

#!/bin/bash
TEXTFILE_DIR="/var/lib/node_exporter/textfiles"
TEMP_FILE=$(mktemp)

# Generate metrics in temp location
{
  echo "# HELP backup_last_success_timestamp Last successful backup time"
  echo "# TYPE backup_last_success_timestamp gauge"
  echo "backup_last_success_timestamp $(date +%s)"
} > "$TEMP_FILE"

# Atomic move prevents partial file reads
mv "$TEMP_FILE" "$TEXTFILE_DIR/backup_status.prom"

Critical Requirement: Use temporary files and atomic moves; writing directly to textfile directory causes metric corruption.

Troubleshooting Common Failures

Memory Usage Explosion

Symptoms: Node Exporter consuming 1GB+ memory
Root Cause: interrupts or slabinfo collectors enabled
Solution:

# Check cardinality
curl -s localhost:9100/metrics | wc -l  # Should be <2000

# Identify problematic collector
curl -s localhost:9100/metrics | grep "^node_" | cut -d'_' -f1,2 | sort | uniq -c | sort -nr

Kubernetes Mount Point Explosion

Symptoms: 500+ filesystem metrics from single node
Root Cause: Docker/Kubernetes overlay mounts not filtered
Solution: Apply mount point exclusion regex (shown above)

Network Interface Cardinality

Symptoms: 100+ network interface metrics on AWS ECS
Solution:

--collector.netdev.device-include="^(eth|ens|eno|enp).*"

Version-Specific Considerations

Version 1.9.1 (Current Recommended)

• Memory leak fixes: Resolved in IRQ pressure collector
• Multiple textfile directories: Supports comma-separated paths
• Improved filtering: URL parameter filtering available

Upgrade Risk Assessment

• High Risk: Cardinality changes between versions can crash Prometheus
• Testing Required: Always test in staging environment first
• Rollback Plan: Backup configurations before upgrade
• Silent Failures: Metric drops after upgrade often go unnoticed

Critical Alerts Configuration

# Essential Node Exporter health monitoring
- alert: NodeExporterDown
  expr: up{job="node-exporter"} == 0
  for: 1m

- alert: NodeExporterHighCardinality
  expr: prometheus_tsdb_symbol_table_size_bytes > 16000000
  for: 5m

- alert: NodeExporterHighMemory
  expr: process_resident_memory_bytes{job="node-exporter"} > 200000000
  for: 5m

Resource Requirements and Planning

Infrastructure Sizing

• CPU: Single core utilization (GOMAXPROCS=1 limitation)
• Memory: 100MB baseline + 2MB per 100 metrics
• Network: 1-5 seconds scrape time with proper filtering
• Storage: Minimal (stateless service)

Scaling Considerations

• Per-node deployment: Required for host-level metrics
• Load balancer support: Possible but complex due to instance-specific metrics
• High availability: Achieved through multiple Prometheus servers, not Node Exporter clustering

Common Misconceptions and Failures

1. "Enable all collectors for complete monitoring" → Causes system crashes and memory exhaustion
2. "Node Exporter monitors containers" → Only monitors host system; use cAdvisor for containers
3. "Default configuration is production-ready" → Default configuration can consume 8GB+ RAM
4. "Binding to 0.0.0.0 is safe on internal networks" → Exposes detailed system information to attackers
5. "More GOMAXPROCS improves performance" → Causes kernel panics on large systems

Essential Documentation References

• GitHub Issues #2530: GOMAXPROCS=1 explanation and kernel panic prevention
• Release Notes v1.9.1: Current version improvements and fixes
• Textfile Collector Scripts: Community-maintained metric collection scripts
• Robust Perception Blog: Performance optimization and cardinality management