Striim: Enterprise Change Data Capture (CDC) Technical Reference

Executive Summary

Striim is an enterprise-grade real-time Change Data Capture (CDC) platform built by ex-Oracle GoldenGate team members. Primary value: log-based CDC that doesn't destroy source database performance while providing sub-second latency for real-time data pipelines.

Critical Success Factors:

• Mission-critical deployments: American Airlines (5,800 daily flights), UPS (package routing), Morrisons (retail operations)
• Built for enterprise scale: handles millions of events per minute
• Schema evolution management prevents Friday 4PM deployment disasters

Technical Architecture & Implementation

Core Technology: Log-Based CDC

WHY IT MATTERS: Only CDC method that doesn't break under production load

• Reads database transaction logs directly (Oracle redo logs, PostgreSQL WAL, SQL Server transaction logs)
• No polling queries hammering source databases
• Maintains transaction integrity and ordering
• Sub-second latency achievable in practice

FAILURE MODES OF OTHER APPROACHES:

• Trigger-based CDC: Adds latency to every transaction
• Query-based CDC: Misses deletes, destroys database performance with SELECT * WHERE timestamp > last_run
• Built-in database CDC: Limited functionality, breaks with schema changes

Database-Specific Implementation Requirements

Oracle Configuration

CRITICAL SETTINGS:

• oracle.net.CONNECT_TIMEOUT=10000 (10 seconds)
• SQLNET.RECV_TIMEOUT=600 (10 minutes)
• Default infinite timeout causes zombie connections consuming Oracle licenses

GOTCHAS:

• Compressed tablespaces require additional setup complexity
• Encrypted tablespaces need special handling
• Connection pooling becomes unstable under high load
• Set max_connections=50 per source to prevent connection exhaustion

PostgreSQL Configuration

CRITICAL SETTINGS:

• max_wal_size: 4GB minimum for production systems
• wal_keep_segments: Monitor religiously to prevent data gaps
• WAL segment recycling will cause irreversible data loss

BREAKING POINTS:

• PostgreSQL 14.2 has known logical replication worker hangs under high load
• pg_resetwal execution destroys CDC continuity
• WAL retention misconfiguration causes permanent data gaps

SQL Server Configuration

CRITICAL WARNING:

• Log file shrinking during maintenance breaks CDC streams
• Configure log backup retention properly
• Transaction log reading stops if logs are truncated

Schema Evolution Management

THE 4PM FRIDAY PROBLEM: Schema changes without notification break most CDC tools

STRIIM'S APPROACH:

• Auto-detection of schema changes
• Three response options: auto-apply, queue for review, halt and alert
• Multi-target handling: different schema adaptations per destination

COMMON FAILURE SCENARIO:

• MySQL ENUM column changes cause parse errors in competing tools
• Striim handles gracefully with configurable response policies

Multi-Target Replication Architecture

CAPABILITY: Single CDC stream feeds multiple destinations simultaneously

• Snowflake (STRING type preference)
• BigQuery (specific field mappings)
• Elasticsearch (custom schemas)
• Kafka clusters
• Legacy systems (fixed-width formats)

RESOURCE IMPACT: No additional load on source database regardless of target count

Performance Specifications & Resource Requirements

Throughput Reality

CLAIMED: "Billions of events per minute"
ACTUAL: Tens of millions of events per minute before hitting infrastructure limits
LIMITING FACTORS: Network I/O, disk I/O, target system capacity (not Striim itself)

Latency Specifications

• Kafka destinations: ~100ms end-to-end
• Snowflake destinations: 2-5 seconds (due to micro-batching)
• BigQuery destinations: Variable based on quotas and slots

Memory Requirements

PRODUCTION REQUIREMENTS:

• Monitor memory usage >80% threshold
• Long-running pipelines leak memory (4.x versions)
• Windowed aggregations consume 8GB+ RAM after several days
• OPERATIONAL REQUIREMENT: Restart pipelines every 2-3 weeks

Infrastructure Scaling

BACKPRESSURE HANDLING: Queues and retries when downstream systems slow
RECOVERY TIME: Serious catch-up time if downstream systems fail for >1 hour
CAPACITY PLANNING: Target systems become bottleneck before Striim

Cost Structure & Budget Planning

Real Production Costs

MINIMUM PRODUCTION BUDGET: $5,000/month for millions of events daily
TYPICAL ENTERPRISE RANGE: $5K-$15K monthly for production workloads
VOLUME PRICING: Six-figure annual commitments required for enterprise discounts

HIDDEN COSTS:

• AWS/Azure data transfer: $1,000+ monthly additional
• Target system capacity (BigQuery slots, Snowflake credits)
• Professional services for complex implementations

Pricing Comparison Matrix

Solution	Monthly Cost	Infrastructure Overhead	Operational Complexity
Striim	$5K-$15K	Managed service	Low
Debezium + Kafka	"Free" + $10K+ infra	High	Very High
Confluent	$8K-$25K	Managed Kafka	Medium
Oracle GoldenGate	$50K+ annually	Oracle licensing	High

Critical Failure Modes & Operational Intelligence

Network & Connection Issues

MOST COMMON FAILURE: Connection drops and zombie connections
MITIGATION:

• Configure connection timeouts properly
• Monitor connection pool health metrics
• Set connection limits per source database

Memory Management Failures

SYMPTOM: OutOfMemoryError: GC overhead limit exceeded
ROOT CAUSE: Memory leaks in long-running streaming applications
SOLUTION: Scheduled pipeline restarts every 2-3 weeks

Target System Integration Failures

BigQuery Specific:

• quotaExceeded errors hit 1,000 queued query limit (not slots)
• Understanding BigQuery slots vs. query quotas is essential
• Data backfill operations can exhaust quotas completely

Schema Registry Dependencies:

• Single point of failure for Kafka-based architectures
• When Schema Registry fails, entire pipeline stops
• Requires high availability configuration

Data Consistency Risks

TRANSACTION BOUNDARY MAINTENANCE: Striim preserves transaction integrity
VERIFICATION REQUIREMENT: Implement data reconciliation between source and targets
VALIDATION APPROACH: Deploy automated consistency checks, especially during first months

Monitoring & Alerting Requirements

Critical Metrics to Track

END-TO-END LATENCY:

• Alert threshold: >10 seconds
• Typical: sub-second to 5 seconds depending on targets

ERROR RATES:

• Alert threshold: >1%
• Track by connector and target system separately

MEMORY USAGE:

• Alert threshold: >80%
• Track per application and per node

CONNECTION HEALTH:

• Database connection pools
• Kafka connection stability
• Schema Registry connectivity

Operational Dashboards

AVOID: Marketing dashboard screenshots
IMPLEMENT:

• Throughput trends (events/second over time)
• Latency histograms by target system
• Error categorization and frequency
• Resource utilization trends

Competitive Analysis & Decision Criteria

When to Choose Striim Over Alternatives

CHOOSE STRIIM IF:

• Budget exists for managed service ($5K+ monthly)
• Team lacks deep Kafka operational expertise
• Schema evolution management is critical
• Multi-database CDC requirements (not just Oracle)
• Visual pipeline management preferred over code

CHOOSE DEBEZIUM + KAFKA IF:

• Strong Kafka operations team available
• Budget constraints require open-source approach
• Comfortable with weekend debugging of distributed systems
• Custom connector development capability exists

CHOOSE CONFLUENT IF:

• Kafka ecosystem commitment already made
• Need broader stream processing beyond CDC
• Comfortable with Kafka complexity but want managed infrastructure

CHOOSE ORACLE GOLDENGATE IF:

• Oracle-only environment
• Existing Oracle DBA expertise
• Budget supports Oracle licensing costs
• Advanced Oracle-specific features required

Technical Capability Comparison

Capability	Striim	Debezium	Confluent	GoldenGate
Oracle CDC Quality	Excellent (ex-GoldenGate team)	Good (community)	Good	Best
Non-Oracle Sources	100+ connectors	Community dependent	Broad ecosystem	Limited
Schema Change Handling	Automated options	Manual custom logic	Manual with Schema Registry	Advanced built-in
Setup Complexity	Low-Medium	High	Medium	High
Operational Support	Enterprise support	Community	Enterprise support	Oracle support

Implementation Roadmap & Risk Mitigation

Phase 1: Proof of Concept (Weeks 1-2)

OBJECTIVES:

• Test with actual production data using free 30-day account
• Validate performance with realistic load
• Test schema change scenarios

RISK MITIGATION:

• Start with non-critical data sources
• Implement monitoring before production deployment
• Document connection and configuration requirements

Phase 2: Production Deployment (Weeks 3-6)

CRITICAL REQUIREMENTS:

• Configure proper database connection settings
• Implement comprehensive monitoring dashboards
• Set up data validation and reconciliation processes
• Plan for operational procedures (pipeline restarts, troubleshooting)

FAILURE PREVENTION:

• Test disaster recovery procedures
• Document escalation procedures for 3AM failures
• Train team on Striim-specific troubleshooting

Phase 3: Scale and Optimize (Month 2+)

SCALING CONSIDERATIONS:

• Monitor target system capacity limits
• Optimize pipeline configurations for throughput
• Implement automated pipeline management

Support & Resources for Implementation

Technical Resources

• Free Developer Account: 30-day full-feature trial
• Getting Started Guide: Wizard-driven setup documentation
• Architecture Documentation: Deployment patterns and scaling considerations
• TQL Reference: Custom transformation SQL variant
• Community Forums: Engineer discussions and troubleshooting

Vendor Support Quality

RESPONSE TIME: Better than typical enterprise software
EXPERTISE LEVEL: Engineers who understand the product architecture
ESCALATION: Some back-and-forth before reaching problem-solving engineers
COMPARISON: Superior to community support, typical of enterprise offerings

Migration Support

GOLDENGATE MIGRATION: Built-in utilities from ex-GoldenGate team
EXPECTATION: Custom logic rewrite required
TEAM TRAINING: Significant retraining investment for operations team

This technical reference provides the operational intelligence needed for informed decision-making about Striim implementation, including real-world performance expectations, cost structures, and failure modes that official documentation typically omits.