Change Data Capture (CDC) Production Implementation Guide

Executive Summary

Change Data Capture (CDC) systems require significant operational investment beyond initial setup. Expect 10-15% database CPU overhead, $200K-1M+ annual costs, and 35-40 hours monthly of manual intervention without proper automation. Critical failure modes include WAL disk exhaustion, connector hangs requiring restarts, and silent schema change breakage.

Architecture Patterns

Kubernetes-Native CDC (Strimzi Operator)

Best For: Container-first organizations
Deployment Complexity: High (initial setup)
Annual Cost: $200K+
Critical Requirements:

• Persistent volume configuration prevents data loss during restarts
• Network policies required for security compliance
• Auto-scaling based on consumer lag, not CPU metrics

Production Configuration:

# Minimum viable Kafka cluster
kafka:
  replicas: 3
  storage:
    type: persistent-claim
    size: 500Gi
    class: fast-ssd
  config:
    min.insync.replicas: 2
    default.replication.factor: 3

Failure Mode: Persistent volumes misconfiguration causes data loss during pod restarts. Symptoms: events missing after cluster restart, WAL position reset.

Managed Cloud CDC (Confluent Cloud)

Best For: Teams prioritizing operational simplicity
Deployment Complexity: Low
Annual Cost: $500K+
Trade-off: 2.5x infrastructure cost vs self-managed for operational simplicity

Traditional VM Deployment

Status: Declining adoption
Best For: Legacy environments with compliance requirements
Operational Overhead: High manual intervention required

Database Integration Specifications

PostgreSQL CDC Requirements

Performance Impact: 10-15% CPU overhead
Critical Monitoring: WAL disk space growth
Breaking Point: >1GB WAL growth in 5 minutes indicates impending failure

Essential Configuration:

-- Replication slot management
slot.name: debezium_production
plugin.name: pgoutput
table.include.list: "public.users,public.orders,public.payments"

Failure Scenarios:

• Tables with capital letters silently break CDC processing
• Connection timeouts require 30-second timeout configuration
• Schema changes without connector restart cause processing failures

MySQL CDC Requirements

Performance Impact: 5-8% CPU overhead with light writes
Critical Monitoring: Binlog position lag
Breaking Point: >60 seconds replication lag

Common Failure: Binlog corruption requires connector offset reset with data loss acceptance.

Event Processing Patterns

Outbox Pattern Implementation

Purpose: Transactional consistency between business logic and event publishing
Critical Requirement: Automatic cleanup prevents table bloat

-- Production outbox table
CREATE TABLE outbox (
    id BIGSERIAL PRIMARY KEY,
    aggregate_type VARCHAR(255) NOT NULL,
    aggregate_id VARCHAR(255) NOT NULL,
    event_type VARCHAR(255) NOT NULL,
    payload JSONB NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT NOW(),
    processed_at TIMESTAMP
);

-- Required cleanup function
CREATE OR REPLACE FUNCTION cleanup_outbox()
RETURNS void AS $$
BEGIN
    DELETE FROM outbox
    WHERE processed_at < NOW() - INTERVAL '7 days';
END;
$$ LANGUAGE plpgsql;

Operational Reality: Without cleanup function, outbox tables consume 40GB+ causing transaction timeouts.

Microservices Event Choreography

Pattern: Services consume CDC events to maintain local views
Critical Insight: Use CDC for views, not synchronous service calls
Failure Mode: Synchronous calls during traffic spikes cause cascade failures

Production Monitoring Requirements

Essential Metrics

Data Freshness: Time between database change and CDC processing
Thresholds:

• Warning: >5 minutes lag
• Critical: >15 minutes lag

Consumer Lag: Messages behind in processing
Thresholds:

• Warning: >10,000 messages
• Critical: >50,000 messages

WAL Growth Rate (PostgreSQL):

• Critical: >1GB growth in 5 minutes
• Indicates connector failure or downstream bottleneck

Business Impact Metrics

Cost Per Event: Infrastructure cost divided by processed events
Typical Range: $0.02-0.05 per event depending on scale
Scale Economics: Larger deployments cost more per event due to operational complexity

Security and Compliance

PII Data Handling

Critical Requirement: Mask PII before CDC capture or face compliance violations

-- Field-level encryption at source
CREATE OR REPLACE FUNCTION encrypt_pii() RETURNS TRIGGER AS $$
BEGIN
    NEW.email = pgp_sym_encrypt(NEW.email, current_setting('app.encryption_key'));
    NEW.ssn = pgp_sym_encrypt(NEW.ssn, current_setting('app.encryption_key'));
    RETURN NEW;
END;
$$ LANGUAGE plpgsql;

GDPR Compliance

Requirements:

• Data classification at source tables
• Environment-specific masking for non-production
• Audit trails for data access
• Right to deletion implementation

Cost Optimization Strategies

Tiered Storage Implementation

Hot Data: 1-hour retention on fast SSDs
Warm Data: 24-hour retention on standard storage
Cold Data: 30-day retention with maximum compression
Savings: 60% storage cost reduction

Auto-Scaling Configuration

Scale Trigger: Consumer lag metrics, not CPU utilization
Stabilization: 5-minute scale-down, 1-minute scale-up windows
Prevents: Resource thrashing during traffic bursts

Failure Scenarios and Recovery

Common Production Failures

WAL Disk Exhaustion:

• Cause: Connector failure without automatic cleanup
• Symptoms: Database writes fail, application errors
• Recovery: Manual WAL cleanup, connector restart
• Prevention: Automated WAL monitoring with 1GB growth alerts

Connector Hangs (CPU 100%):

• Cause: Unknown Debezium issue
• Recovery: Connector restart resolves issue
• Operational Response: Automated restart cron job implemented

Schema Change Breakage:

• Cause: Database schema changes without connector coordination
• Symptoms: Processing stops, no obvious errors
• Recovery: Connector restart with schema refresh
• Prevention: Schema change procedures including CDC testing

Disaster Recovery Requirements

RTO: 15 minutes for connector restart
RPO: 5 minutes maximum data loss acceptable
Dependencies: Database availability, Kafka cluster health, schema registry

Resource Requirements

Infrastructure Sizing

Kafka Memory: 8GB base + 25% of storage size for page cache
Connect Workers: 1 worker per 10 connectors, minimum 2 for HA
Storage Growth: 50GB per partition for <100K events/day, 500GB for >1M events/day

Human Resources

Engineering Time: 35-40 hours monthly for operational maintenance
Expertise Required: Database administration, Kafka operations, Kubernetes knowledge
On-call Requirements: 24/7 coverage for production CDC systems

Implementation Decision Criteria

Use CDC When:

• Sub-minute data freshness required
• Multiple downstream systems need identical data changes
• Building event-driven architectures
• Compliance requires real-time data lineage

Avoid CDC When:

• 15+ minute update latency acceptable (use batch ETL)
• <1000 database changes daily
• Team lacks distributed systems experience
• Primary use case is data warehousing

Technology Selection Matrix

Pattern	Deployment Complexity	Annual Cost	Operational Overhead	Best For
Kubernetes Native	High	$200K+	Medium	Container-first orgs
Managed Cloud	Low	$500K+	Low	Operational simplicity
Traditional VM	Medium	Variable	High	Legacy compliance
Hybrid Cloud	Very High	$1M+	Very High	Multi-region compliance

Common Misconceptions

"CDC is Real-time": Typical latency 30 seconds to 5 minutes under normal conditions
"CDC Reduces Database Load": Actually increases CPU 10-15% and requires additional connections
"Set and Forget": Requires ongoing operational attention, monitoring, and maintenance
"Vendor Demos Represent Reality": Production implementations require significant additional configuration

Success Metrics

Technical Success:

• <5 minute average data freshness
• <2 unplanned outages per quarter
• <$0.05 cost per processed event

Operational Success:

• Documented runbooks for all failure scenarios
• Automated monitoring and alerting
• Quarterly disaster recovery testing
• Team expertise in CDC operations

Critical Dependencies

Database Administration: CDC changes database configuration and monitoring requirements
Platform Engineering: Kubernetes operators, persistent storage, networking
Security Team: PII handling, compliance validation, audit requirements
Application Teams: Event schema design, consumer implementation, error handling