etcd: AI-Optimized Technical Reference

Core Function and Critical Constraints

etcd is a distributed key-value store using Raft consensus that serves as Kubernetes' sole datastore option. Critical failure mode: When etcd breaks, kubectl becomes non-functional and entire clusters enter read-only state.

Data Storage Scope:

• Pod locations and state
• Service endpoints and load balancer configs
• Secrets, ConfigMaps, RBAC rules
• Resource quotas and network policies

Production Configuration Requirements

Storage Requirements (Non-Negotiable)

• Disk type: NVMe SSDs mandatory - spinning disks cause constant leader elections
• Write latency threshold: <10ms (anything over 10ms = leader election chaos)
• Storage isolation: Dedicated volumes required - shared storage causes write spikes during other operations
• AWS specifics: GP3 with provisioned IOPS or local NVMe only - GP2 volumes will fail

Hardware Specifications

• Memory: 512MB minimum, monitor for growth to 2GB+ (v3.5 and earlier had memory leaks)
• Network latency: <50ms round-trip between nodes for stable operation
• Cluster size: 3, 5, or 7 nodes only (odd numbers for majority consensus)

Performance Thresholds and Breaking Points

• Write performance: ~10K writes/sec maximum on optimal hardware
• Storage limits: Performance degrades significantly after 2GB
• Failover time: 5-10 seconds during leader elections (no writes during this period)
• Scale limits: Struggles at 1000+ Kubernetes nodes due to constant pod update churn

Critical Failure Scenarios and Recovery

Network Partition Behavior

• Only majority partition remains writable (2/3 or 3/5 nodes)
• Minority partition becomes read-only
• Consequence: Better complete write failure than inconsistent state

Common Production Failures

1. Certificate expiration: Entire cluster communication stops
2. Disk space exhaustion: Cluster becomes read-only, kubectl fails
3. Memory leaks (pre-v3.6): Gradual memory growth to 2GB+ over weeks
4. Storage latency spikes: Automatic leader re-elections, write timeouts

Disaster Recovery Requirements

• Backup method: etcdctl snapshot save backup.db
• Restoration gotcha: New cluster IDs break existing client connections
• Downtime requirement: Full cluster downtime required for restoration
• Testing mandate: Monthly restore testing to verify backup integrity

Monitoring and Health Indicators

Critical Metrics (Set Alerts)

• etcd_server_is_leader flapping = leader election chaos
• etcd_disk_wal_fsync_duration_seconds >100ms = storage problems
• etcd_mvcc_db_total_size_in_bytes approaching 2GB = quota/cleanup needed
• etcd_network_peer_round_trip_time_seconds spiking = network issues

Memory Usage Patterns

• v3.6: Fixed most memory leaks, 50% reduction in usage
• Watch for: Compaction failures, excessive watch connections, large value storage

Version-Specific Intelligence

etcd v3.6 Improvements

• Memory leak fixes: 50% reduction in memory usage
• Robustness testing: Jepsen-style testing uncovered and fixed data inconsistency bugs
• Downgrade support: Can roll back to v3.5 without cluster rebuild
• Verdict: First version suitable for production without extensive testing

Pre-v3.6 Issues

• Memory leaks requiring regular monitoring and restarts
• Data corruption scenarios in specific failure modes
• Compaction failures causing memory explosion

Security Configuration Reality

TLS Requirements

• All cluster communication requires TLS in production
• Failure mode: Certificate expiration = complete cluster failure
• Operational requirement: 30-day expiration warnings mandatory
• Certificate rotation requires careful timing to avoid downtime

Use Case Suitability Analysis

Appropriate Uses

• Kubernetes cluster state (mandatory)
• Service discovery with strong consistency requirements
• Distributed locking for critical operations
• Configuration requiring immediate consistency

Inappropriate Uses

• Application databases (use PostgreSQL instead)
• High-volume data storage (2GB practical limit)
• Eventually consistent scenarios (unnecessarily restrictive)

Comparative Analysis with Alternatives

vs ZooKeeper

• Advantage: No JVM heap tuning nightmare
• Disadvantage: Same network partition write unavailability
• Migration factor: Simpler operational model

vs Redis Cluster

• Advantage: Strong consistency prevents phantom states
• Disadvantage: Write unavailability during partitions vs Redis eventual consistency
• Performance: Redis faster but lies about data freshness

vs DynamoDB

• Advantage: On-premises deployment control
• Disadvantage: Manual operational overhead vs AWS managed service
• Cost: Predictable vs AWS billing surprises

Financial Services Specific Requirements

Why Banks Use etcd

• Strong consistency prevents phantom trades and double-execution
• ACID compliance for regulatory requirements
• MVCC provides bulletproof audit trails
• Fail-fast behavior preferred over inconsistent success

Hidden Infrastructure Costs

• Odd-numbered cluster requirements increase hardware needs
• Cross-datacenter latency requires dedicated fiber connections
• Disaster recovery setup more expensive than anticipated

Resource Investment Requirements

Time Investments

• Initial setup: 1-2 weeks for proper production configuration
• Ongoing monitoring: Daily metric review mandatory
• Disaster recovery testing: Monthly restore validation required
• Certificate management: Quarterly rotation planning

Expertise Requirements

• Distributed systems understanding for troubleshooting
• Storage performance analysis skills
• Network latency debugging capabilities
• Kubernetes integration knowledge for production use

Infrastructure Costs

• Premium storage required (NVMe SSDs, high IOPS)
• Dedicated infrastructure for each cluster member
• Network quality requirements increase connectivity costs
• Monitoring and alerting system integration overhead

Troubleshooting Decision Trees

Performance Issues

1. Check disk latency first (iostat -x 1)
2. Verify network latency between members
3. Monitor compaction success rate
4. Check for quota limits approaching

High Availability Issues

1. Verify odd-numbered cluster configuration
2. Check network partition simulation capability
3. Validate certificate expiration schedules
4. Test failover procedures under load

Memory/Resource Issues

1. Identify version (v3.6+ preferred)
2. Monitor watch connection counts
3. Check compaction frequency and success
4. Analyze stored value sizes

Breaking Points and Scalability Limits

Hard Limits

• 2GB storage before significant performance degradation
• 1000+ Kubernetes nodes = constant leadership churn
• 10ms+ disk latency = unusable leader election behavior
• 50ms+ network RTT = cross-datacenter deployment failure

Soft Limits

• 10K writes/sec theoretical maximum on optimal hardware
• Real-world performance significantly lower under Kubernetes load
• Memory usage growth over time requires periodic monitoring

Operational Red Flags

Immediate Action Required

• Leader election messages in logs
• Certificate expiration within 30 days
• Database size approaching 1.8GB
• Disk write latency spikes above 50ms

Planning Required

• Memory usage trend toward 1.5GB
• Kubernetes cluster growth past 800 nodes
• Network maintenance affecting inter-node communication
• Storage performance degradation trends

This reference provides decision-support information for etcd deployment, scaling, and maintenance while preserving all operational intelligence from real-world production experience.