ELK Stack for Microservices Logging - AI-Optimized Knowledge Base

Technology Overview

ELK Stack Components: Elasticsearch (distributed search engine), Logstash (log processing pipeline), Kibana (visualization interface), Filebeat/Beats (log collection agents)

Data Flow Architecture: Raw application logs → Beats/Logstash (collection & parsing) → Elasticsearch (indexing & storage) → Kibana (visualization & search)

Critical Failure Point: When any component in the chain fails, the entire logging pipeline breaks

Configuration That Actually Works

Elasticsearch Production Settings

Memory Configuration:

• Heap size: 50% of system RAM, never exceed 30.5GB
• Sweet spot: 26-30GB heap on 64GB machines
• Above 32GB: Loss of compressed ordinary object pointers (compressed oops) optimization
• Performance threshold: Above 26GB loses zero-based compressed oops

ES_JAVA_OPTS: "-Xms8g -Xmx8g"  # Same min/max values required
# NOT: "-Xms32g -Xmx32g" on 64GB machine - causes excessive GC pauses

Resource Requirements (Production-Tested):

replicas: 3
minimumMasterNodes: 2
resources:
  requests:
    memory: "8Gi"  # Start small, scale up
  limits:
    memory: "16Gi"  # Not 32Gi as documentation suggests
heap:
  max: "8g"  # Never exceed 50% of container memory

Cluster Health States:

• Green: All primary and replica shards allocated
• Yellow: Missing replica shards but functional
• Red: Missing primary shards, data loss imminent

Network Partition Prevention:

• Use odd number of master nodes
• Set minimum_master_nodes to (total_masters / 2) + 1
• Prevents split-brain syndrome and data loss

Storage Cost Optimization

Storage Strategy:

• Hot data (7-14 days): NVMe SSD - $15K/month for high-performance access
• Warm data (older logs): SATA drives - saves $8K/month compared to all-NVMe setup
• Start with 1TB NVMe for hot indices

Disk Space Management:

• Low watermark: 85% disk usage
• High watermark: 90% disk usage
• Flood stage: 95% - triggers read-only mode with error: cluster_block_exception: blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]

Application Integration Patterns

Pattern	Production Reliability	Cost Impact	Primary Failure Mode
Direct Integration	High failure rate	Low cost	App crashes when ES down
Filebeat + Logstash	Most common production	Medium cost	Logstash config/heap issues
Kafka + ELK	Enterprise grade	High cost	Kafka rebalances/ES heap
Sidecar Container	K8s recommended	200MB RAM per pod	Pod memory limits
Agent-Based	Maintenance intensive	Low cost	Agent version conflicts

Critical Failure Scenarios and Solutions

Elasticsearch Cluster Red Status

Root Cause Priority:

1. Disk space exhaustion (95%+ usage)
2. Memory issues (OOM heap space)
3. Unassigned shards
4. Network partitions

Diagnostic Commands:

# Always check disk space first
curl -X GET "localhost:9200/_cat/allocation?v&h=shards,disk.indices,disk.used,disk.avail,disk.percent,host"

# Cluster health details
curl -X GET "localhost:9200/_cluster/health?pretty"

# Find specific shard problems
curl -X GET "localhost:9200/_cat/shards?v&h=index,shard,prirep,state,unassigned.reason"

Memory-Related Failures

OutOfMemoryError Symptoms:

• Error: java.lang.OutOfMemoryError: Java heap space
• Cluster immediately goes red
• Excessive GC pauses (>10% CPU time)

Prevention:

• Monitor GC time with jstat -gc [PID]
• Keep GC time under 10% of CPU time
• Use identical min/max heap settings

Logstash Pipeline Failures

Common Failure Indicators:

• Pipeline throughput drops to 0 events/second
• Input events: 0 (source connection broken)
• Output events: 0 (Elasticsearch down/rejecting documents)

Diagnostic API:

curl localhost:9600/_node/stats/pipelines?pretty

Configuration Pitfalls:

# BREAKS EVERYTHING - missing closing brace
filter {
  if [field] == "value"  # Missing closing brace
    mutate { add_tag => "broken" }
  }
}

# WORKS - proper YAML syntax
filter {
  if [field] == "value" {
    mutate { add_tag => ["working"] }
  }
}

Search Performance Issues

Mapping Problems:

• Using text fields for exact matching (causes slow queries)
• Missing keyword fields for aggregations
• Incorrect field types for time-based queries

Optimal Field Mapping:

{
  "properties": {
    "service_name": { "type": "keyword" },    # Exact matches
    "message": { "type": "text" },           # Full-text search
    "timestamp": { "type": "date" }          # Time range queries
  }
}

Resource Requirements and Costs

Real-World Resource Consumption

Elasticsearch Cluster (Production):

• 3-node cluster: 24GB RAM per node minimum
• Storage: 1TB NVMe for hot data + cheaper SATA for cold
• CPU: 8-16 cores per node (I/O bound workload)
• Network: 10Gbps for inter-node communication

Logstash Resource Usage:

• CPU intensive: 4-8 cores minimum
• Memory: 8-16GB heap + system memory
• JVM tuning required for production loads

Filebeat Overhead:

• Sidecar pattern: 200MB RAM per pod
• Minimal CPU impact
• Disk I/O dependent on log volume

Cost Optimization Strategies

Storage Costs (Based on real deployments):

• All-NVMe approach: $15K/month (enterprise workload)
• Hot/warm strategy: $7K/month (65% cost reduction)
• ILM implementation: Additional 30-50% savings

Index Lifecycle Management (ILM):

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": { "actions": { "rollover": { "max_size": "50gb", "max_age": "7d" }}},
      "delete": { "min_age": "30d", "actions": { "delete": {} }}
    }
  }
}

Version-Specific Critical Issues

Elasticsearch 8.1.3

• Critical Bug: Memory leak in ingest pipeline processor
• Impact: Gradual heap exhaustion leading to OOM
• Solution: Skip version, upgrade to 8.2+

Upgrade Risks

• Index mapping incompatibilities between major versions
• Plugin compatibility breaks
• Performance regression in specific configurations
• Mitigation: Test upgrades on production data copy, not demo data

Production Monitoring Requirements

Essential Alerts

Critical Thresholds:

• Elasticsearch cluster status != green for >2 minutes
• Disk usage >85% (not 90% - too late at 90%)
• Logstash pipeline throughput drops 50%
• JVM heap usage >75%
• Index creation failures (indicates mapping problems)

Monitoring Tools

Recommended Stack:

• Cerebro: Better than built-in Elasticsearch monitoring
• ElastAlert2: Reliable alerting without vendor lock-in
• Prometheus + Grafana: Comprehensive metrics collection

Kibana Dashboard Backup:

# Export dashboards before they disappear
curl -X GET "localhost:5601/api/saved_objects/_export" \
  -H 'Content-Type: application/json' \
  -H 'kbn-xsrf: true' \
  -d '{"type":"dashboard"}'

Security Implementation

Minimal Security Configuration

TLS and Authentication:

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

Alternative: NGINX reverse proxy with basic auth

• Often easier to implement than X-Pack security
• Lower complexity, fewer failure points
• Sufficient for most production environments

Security Failures

• Cryptolocker attacks on exposed Elasticsearch clusters
• Data exfiltration through unsecured Kibana instances
• Resource abuse from public access to cluster

Integration Architecture Patterns

Kafka Buffer Implementation

Why Kafka Buffer:

• Prevents log loss during Elasticsearch downtime
• Handles traffic spikes without pipeline failures
• Provides data durability and replay capability

Configuration:

# Filebeat to Kafka
output.kafka:
  hosts: ["kafka1:9092", "kafka2:9092", "kafka3:9092"]
  topic: 'logs-%{[service.name]}'
  required_acks: 1
  compression: gzip

Application Logging Best Practices

JSON Logging Configuration:

logging:
  pattern:
    console: "%d{yyyy-MM-dd HH:mm:ss} [%thread] %-5level [%X{traceId:-}] %logger{36} - %msg%n"

Correlation ID Implementation:

spring:
  sleuth:
    sampler:
      probability: 0.1  # Don't trace everything - performance impact
    zipkin:
      enabled: false   # Only if Zipkin actually running

Troubleshooting Decision Trees

Cluster Red Status

1. Check disk space (_cat/allocation)
2. If disk OK → Check unassigned shards (_cat/shards)
3. If shards OK → Check heap usage and GC time
4. If memory OK → Check network connectivity between nodes

Performance Degradation

1. Check field mappings for proper types
2. Verify query patterns (avoid wildcard prefixes)
3. Check index size and shard distribution
4. Monitor resource utilization (CPU, memory, I/O)

Data Loss Prevention

1. Implement ILM policies before disk fills
2. Monitor cluster health continuously
3. Backup critical indices regularly
4. Test restore procedures

Real-World Deployment Lessons

Common Misconceptions

• "Elasticsearch is a database": It's a search engine; treat it as such
• "More heap = better performance": Above 32GB actually decreases performance
• "Official Helm charts are production-ready": They're resource-intensive and often over-engineered

Time Investment Requirements

• Initial setup: 2-3 weeks for production-grade deployment
• Performance tuning: 1-2 weeks of iterative optimization
• Monitoring setup: 1 week for comprehensive alerting
• Team training: 1 month for operational proficiency

Expertise Prerequisites

• Strong understanding of JVM tuning and garbage collection
• Kubernetes knowledge for containerized deployments
• Linux system administration for performance optimization
• Network troubleshooting for cluster communication issues

This knowledge base provides actionable intelligence for successful ELK Stack implementation while highlighting critical failure modes and their prevention strategies.