Debezium CDC: Production Implementation Guide

Critical Configuration Requirements

Database Prerequisites

• PostgreSQL: wal_level=logical, max_replication_slots=10 minimum
• MySQL: binlog_format=ROW, binlog_row_image=FULL, binlog_expire_logs_seconds=604800 (7 days)
• Oracle: Supplemental logging enabled: ALTER DATABASE ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;
• SQL Server: CDC enabled at database and table level
• MongoDB: Replica set required (single node will fail)

Infrastructure Requirements

• Kafka Cluster: 5 brokers minimum (3 insufficient for production headroom)
• Replication Factor: 3 with min.insync.replicas=2
• Memory: 8GB heap minimum per connector (2GB default causes GC failures)
• Setup Time: 3 weeks for Kafka + Debezium (not 3 hours as documentation suggests)

Production Deployment Modes

Mode	Use Case	Fault Tolerance	Setup Complexity	Production Viability
Kafka Connect	Production standard	High (survives node failures)	5 days	✅ Recommended
Debezium Server	Standalone without Kafka	None	2 weeks trial period	❌ Abandoned after 2 weeks
Embedded Engine	Java library	None	High	❌ Memory leaks at 3am

Performance Thresholds

Latency Expectations

• Normal Operations: Sub-second latency
• Under Load: Can spike to minutes during connector failures
• Recovery Time: 48 hours for 500GB table full snapshot
• Critical Threshold: Monitor lag > 60 seconds (alert immediately)

Breaking Points

• UI Failure: 1000+ spans make debugging distributed transactions impossible
• Volume Spike: 50x increase requires partition scaling (3→12 partitions)
• Memory Leak: Default 2GB heap causes frequent restarts

Critical Failure Scenarios

Schema Evolution (Silent Killer)

• Failure Case: NOT NULL column without default crashes connector during snapshot
• Impact: 3-hour pipeline downtime
• Prevention: Coordinate schema changes with connector restarts
• Process: Backwards-compatible changes only, test environment first, plan downtime

Offset Loss

• Trigger: Lost __connect-offsets topic or corruption
• Impact: Full snapshot restart (48+ hours for large tables)
• Recovery: No automated recovery, manual full resync required
• Prevention: Regular offset topic backups

Replication Slot Failures (PostgreSQL)

• Cause: Connector down longer than slot retention
• Symptom: "replication slot does not exist" error
• Disk Impact: Slots fill disk if connector stops consuming
• Recovery: Manual slot recreation, potential data gap

Binlog Position Loss (MySQL)

• Trigger: Binlog rotation during connector downtime
• Impact: Complete data resync required
• Prevention: binlog_expire_logs_seconds=604800 minimum
• Recovery Time: Full snapshot duration (hours to days)

Tool Comparison Matrix

Solution	Real Cost/Month	Setup Complexity	Reliability	Documentation Quality
Debezium	$500 (infrastructure)	High (Kafka required)	Good when tuned	Decent
AWS DMS	$2-5k	Medium	AWS-dependent	AWS-grade
Oracle GoldenGate	$$$$$+ (mortgage-level)	Nightmare	Excellent	Oracle-grade
Airbyte	$200-2k+ (freemium trap)	Low	Hit or miss	Pretty good
Striim	Enterprise pricing	Medium	Usually works	Enterprise-grade

Production Use Case Failures

Microservices Data Sync

• Failure: Schema changes break outbox pattern
• Root Cause: Forgot to update outbox schema after column addition
• Duration: 6 hours debugging (connector didn't crash, silently ignored)
• Fix: Schema change coordination process required

Real-Time Analytics

• Failure: 20-minute lag during marketing campaign
• Cause: 50x volume spike overwhelmed 3-partition setup
• Solution: Horizontal scaling to 12 partitions
• Lesson: Load test CDC pipeline, monitor consumer lag religiously

Search Index Sync

• Failure: Elasticsearch overwhelmed during 4-hour event replay
• Cause: Bulk indexing feedback loop after maintenance window
• Fix: Implement backpressure and circuit breakers
• Impact: Downstream system degradation affects entire pipeline

Cache Invalidation

• Complexity: Tracking relationships across multiple tables
• Solution: Abandoned surgical invalidation for 5-minute TTL
• Lesson: Simple solutions often beat complex ones

Monitoring Requirements

Critical Metrics

• Connector lag (alert > 60 seconds)
• Connector status (running/failed)
• Memory usage (alert > 80%)
• Database connection health

Infrastructure Stack

• Monitoring: Prometheus + Grafana
• Metrics Export: JMX via Kafka Connect
• Alerting: Connector failures and performance degradation

Common Production Issues

Random Connector Restarts

• Root Cause: Memory leaks with default 2GB heap
• Solution: 8GB minimum heap size
• Symptom: OutOfMemoryError: Java heap space in logs

Events Not Flowing

• Check: Actual database changes occurring
• Verify: Connector status via REST API
• Common Causes: Database permissions, network issues, schema registry down

Offset Flush Failures

• Error: "Failed to flush offsets to storage"
• Cause: Kafka cluster unreachable or insufficient brokers
• Fix: Increase offset.flush.timeout.ms to 60000ms

Oracle LogMiner Crashes

• Requirements: Supplemental logging enabled
• Issues: Memory consumption, redo log archival speed
• Solution: Memory tuning and proper logging configuration

Resource Investment Reality

Time Requirements

• Kafka Setup: 3 weeks (not hours)
• Connector Configuration: 5 days for production-ready setup
• Schema Change Process: Manual coordination required
• Recovery Operations: 48+ hours for large table snapshots

Expertise Requirements

• Kafka Administration: Essential for troubleshooting
• Database Administration: Required for log configuration
• JVM Tuning: Necessary for memory optimization
• Monitoring Setup: Critical for operational visibility

Decision Criteria

Choose Debezium When

• Already have Kafka infrastructure
• Need open-source CDC solution
• Can invest in Kafka expertise
• Require horizontal scaling

Avoid Debezium When

• No Kafka experience
• Need immediate production deployment
• Cannot tolerate 3-week setup timeline
• Require 24/7 enterprise support

Bottom Line Assessment

After 2 years production experience: Best available CDC solution, but "best" reflects poor state of CDC market. Saved hundreds of hours of manual syncing, prevented countless stale data bugs. Expect earning reliability through debugging, memory tuning, and monitoring setup. Budget 3 weeks setup time, not 3 hours.