AWS Lambda DynamoDB: Production-Ready Integration Guide

Architecture Overview

Core Components:

• DynamoDB stores data with automatic stream capture
• DynamoDB Streams capture all data changes (INSERT, UPDATE, DELETE)
• Lambda processes stream events in near real-time
• 1:1:1 mapping between DynamoDB partitions, stream shards, and Lambda functions

Performance Specifications:

• DynamoDB latency: 2-5ms average (not sub-millisecond as marketed)
• Stream retention: 24 hours maximum
• Lambda cold starts: 5-10% in production (not <1% as claimed)
• Default Lambda concurrency limit: 1,000 concurrent executions

Critical Failure Modes

Hot Partition Bottlenecks

Problem: Single busy partition becomes chokepoint for entire pipeline
Impact: Complete system failure for 3+ hours during traffic spikes
Root Cause: 1:1:1 mapping means hot partitions process sequentially
Solution: Design partition keys with high cardinality (e.g., userId#timestamp vs userId)

Stream Processing Failures

Problem: Iterator age spikes to 2+ hours without warning
Impact: Loss of "real-time" processing capability
Root Cause: DynamoDB adaptive capacity delays during traffic spikes
Solution: Monitor iterator age; alert when >30 seconds

Poison Pill Records

Problem: Single malformed record blocks entire shard for 24 hours
Impact: Complete processing halt until manual intervention
Root Cause: Default infinite retry behavior
Solution: Set MaximumRetryAttempts: 3 + enable BisectBatchOnFunctionError

Production Configuration

EventSourceMapping Settings

{
  "BatchSize": 1000-5000,           // Default 100 is too small
  "MaximumBatchingWindowInSeconds": 5,  // Reduces invocations by 90%
  "ParallelizationFactor": 2-4,     // Increases throughput but multiplies cold starts
  "MaximumRetryAttempts": 3,        // Prevents infinite retry hell
  "BisectBatchOnFunctionError": true, // Isolates poison pills
  "MaximumRecordAgeInSeconds": 300  // Discard old records
}

Error Handling Pattern

exports.handler = async (event) => {
    const failures = [];
    
    for (const record of event.Records) {
        try {
            await processRecord(record);
        } catch (error) {
            console.error(`Record ${record.dynamodb.SequenceNumber} failed:`, error);
            
            // Don't retry network timeouts or 5xx errors
            if (error.name === 'TimeoutError' || error.statusCode >= 500) {
                failures.push({ recordId: record.dynamodb.SequenceNumber });
            } else {
                await sendToDLQ(record, error);
            }
        }
    }
    
    return { batchItemFailures: failures };
};

Memory Optimization

• 512MB-1GB: Optimal for most stream processing
• Memory = CPU = Performance = Cost: Higher memory often reduces total cost
• ARM64: Better price/performance but NPM package compatibility issues

Cost Analysis

Pricing Breakdown (100k user app example)

• Monthly cost: ~$200
• Lambda billing change (August 2025): +20-30% due to initialization phase billing
• Batch size impact: 10k records vs 100 records = $50/month vs $500/month

Cost Optimization Strategies

• Batch Size: Use 1,000-5,000 records minimum
• Batching Window: 5 seconds reduces invocations dramatically
• Memory Allocation: Use Lambda Power Tuning tool for optimization
• DynamoDB: On-demand costs 5x more but scales automatically

Monitoring Requirements

Critical Metrics

• Iterator Age >30 seconds: Immediate alert required
• Error Rate >1%: System degradation
• Duration Increase: Check for cold starts or downstream latency
• ConcurrentExecutions: Monitor for limit breaches

Debugging Tools

• X-Ray: Actually works for stream processing (unlike most AWS services)
• CloudWatch Query: filter @type = "REPORT" | stats sum(@initDuration) by bin(5m)
• Structured Logging: Use correlation IDs matching DynamoDB items

Data Consistency Guarantees

What Works

• Item-level ordering: Changes to same item always arrive in order
• 24-hour retention: Built-in resilience against temporary failures
• Audit logs: Stream records include old and new values

What Doesn't Work

• Cross-item coordination: Streams don't guarantee ordering across items
• Global Tables: Last writer wins conflict resolution
• Real-time guarantees: Iterator age spikes break real-time processing

Common Integration Patterns

Reliable Patterns

1. Cache Invalidation: Update Redis/ElastiCache on data changes
2. Search Indexing: Push to Elasticsearch automatically
3. Audit Logging: Track all data changes with old/new values
4. Event Publishing: Trigger EventBridge for microservices

Anti-Patterns to Avoid

• Complex Queries: NoSQL query limitations
• High-frequency Events: Page view tracking will destroy budget
• Data Warehouse Replacement: Not suitable for analytical workloads
• ACID Transactions: Limited to single-item operations

Comparison Matrix

Aspect	Lambda + DynamoDB	Lambda + RDS Proxy	Step Functions + DynamoDB	EventBridge + Lambda
Operational Overhead	✅ No servers	❌ RDS maintenance	❌ Complex debugging	✅ No servers
Performance	⚠️ 2-5ms	❌ 20-100ms	❌ 500ms+	⚠️ Variable delays
Scaling	⚠️ Hot partition limits	❌ Connection pools	✅ Unlimited	⚠️ Surprise limits
Cost	Variable	High from start	Expensive transitions	Event costs accumulate
Data Consistency	Item-level only	Full ACID	Eventually consistent	Hope and pray
Real-time	When working	Polling only	Not real-time	"Near" real-time
Complex Queries	NoSQL limitations	SQL works	Lambda spaghetti	No queries
Error Handling	When configured	Connection failures	Works well	Event replay complex
Cold Start Impact	5-10% reality	Connection delays	Every invocation	Every event

Troubleshooting Guide

Iterator Age Spiking

1. Check CloudWatch for Lambda errors
2. Set MaximumRetryAttempts to 3
3. Enable BisectBatchOnFunctionError
4. Increase Lambda memory if no errors
5. Check ConcurrentExecutions limits

Hot Partition Throttling

• Immediate: Implement write sharding with random suffixes
• Long-term: Redesign partition keys for better distribution
• Pattern: Use userId#timestamp instead of userId

Lambda ECONNREFUSED Errors

1. Check IAM permissions (usually the cause)
2. Verify VPC configuration if applicable
3. Test network connectivity
4. Add missing service permissions

Stream Processing Stops

1. Restart EventSourceMapping (disable/enable)
2. Increase Lambda memory temporarily
3. Reduce batch size for faster processing
4. Check account limits

Cold Start Cost Impact

• Query: filter @type = "REPORT" | stats sum(@initDuration) by bin(5m)
• Solutions: SnapStart for Java/Python, Provisioned Concurrency for sustained traffic
• Budget: Expect 20-30% cost increase from initialization billing

Resource Requirements

Development Time

• Simple implementation: 1-2 days
• Production-ready with error handling: 1-2 weeks
• Complex multi-region setup: 1-2 months

Expertise Required

• Basic: Understanding of NoSQL access patterns
• Intermediate: Lambda configuration and monitoring
• Advanced: DynamoDB partition design and stream optimization

Infrastructure Dependencies

• Required: DynamoDB table with streams enabled
• Recommended: CloudWatch alarms, X-Ray tracing, Dead Letter Queues
• Optional: Lambda Provisioned Concurrency for latency-sensitive apps

Breaking Points and Limits

Hard Limits

• Stream retention: 24 hours maximum
• Lambda timeout: 15 minutes maximum
• Item size: 400KB maximum (affects audit logs)
• Batch size: 10,000 records maximum

Practical Limits

• Concurrency: 1,000 default (request increases take days)
• Throughput: 1,000-10,000 records/second per shard
• Hot partitions: Single partition becomes bottleneck
• Cold starts: 5-10% of invocations in production

Security Considerations

IAM Requirements

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeStream",
        "dynamodb:GetRecords",
        "dynamodb:GetShardIterator",
        "dynamodb:ListStreams"
      ],
      "Resource": "arn:aws:dynamodb:region:account:table/table-name/stream/*"
    }
  ]
}

Best Practices

• Never log sensitive data in Lambda functions
• Use parameter store for configuration secrets
• Implement least privilege access
• Enable VPC endpoints for private connectivity

Migration Strategies

From Polling to Streams

1. Implement stream processing in parallel
2. Compare outputs for consistency
3. Gradually reduce polling frequency
4. Switch traffic and remove polling

Blue-Green Deployments

1. Disable EventSourceMapping
2. Deploy new function version
3. Re-enable mapping with new version
4. Monitor for errors during transition

Multi-Region Setup

• Each region has separate streams
• Use Lambda in each region for local processing
• Forward to central queue/EventBridge for global coordination
• Handle Global Tables conflict resolution