AWS Developer Tools: Build Optimization & Cost Control Reference

Critical Performance Bottlenecks

Instance Right-Sizing Impact

• Default Problem: Teams use t3.large by default, burning 60% more money
• Cost Reality: BUILD_GENERAL1_SMALL ($0.005/min) vs BUILD_GENERAL1_MEDIUM ($0.01/min)
• Real Impact: 10-minute build, 50 runs daily = $125/month vs $250/month ($1,500 yearly waste)
• Decision Criteria: CPU usage consistently under 50% = downsize, memory usage 80%+ = upsize
• Monitoring Requirement: Use CloudWatch metrics for 30 days before right-sizing

Caching Configuration That Actually Works

S3 vs Local Caching Performance

• Break-even Point: 50MB+ dependencies benefit from S3 caching
• Local Cache Advantage: Faster than S3 for repeated builds in same region
• Critical Limitation: Local cache gets randomly nuked - cannot depend on it
• Latency Reality: S3 caching adds overhead for small dependencies

Production Cache Configuration:

cache:
  type: LOCAL
  modes:
    - LOCAL_DOCKER_LAYER_CACHE
    - LOCAL_SOURCE_CACHE

Real Results: Node.js builds reduced from 12 minutes to 4 minutes, $800/month to $320/month

Custom Docker Images vs Standard Images

• Performance Gain: 8-minute builds reduced to 3 minutes
• Cost: Maintenance overhead - requires monthly security updates
• Failure Mode: Outdated images create security vulnerabilities
• Implementation: Bake Python, Node.js, dependencies into custom ECR image

Cost Control Strategies

Hidden Cost Killers

S3 Artifact Storage Bleeding

• Problem: CodeBuild dumps everything to S3 by default
• Cost Impact: $200/month from forgotten build artifacts
• Solution: Lifecycle policies + selective artifact upload

artifacts:
  files:
    - 'dist/**/*'  # Only upload what you actually need

Data Transfer Cost Reality

• Cross-Region Impact: $0.09/GB for Docker image transfers
• Real Example: 2GB image, 100 daily pulls across regions = $540/month
• Solution: Keep ECR repositories in same region as CodeBuild

Build Frequency Optimization

• Monorepo Disaster: Full rebuilds for single file changes
• Success Case: Team reduced costs from $1,200 to $400/month (67% reduction)
• Implementation: Change detection with Rush/Lerna for affected packages only

Regional Cost Arbitrage

• Savings: US East (Virginia) typically 10-15% cheaper than US West
• Hidden Cost: Data transfer back to deployment region may exceed savings
• Decision Rule: Calculate total cost including transfers before switching regions

Performance vs Cost Trade-offs

Strategy	Speed Impact	Cost Savings	Complexity	Maintenance Overhead
Right-size instances	Neutral/slower	60% reduction	Low	None
S3 caching	50% faster	Mixed	Medium	Cache key management
Custom images	3x faster startup	Neutral	High	Monthly security updates
Parallel stages	40% faster	Higher cost	High	Resource contention issues
Build frequency limits	Slower feedback	40% reduction	Low	None

Critical Failure Modes

Common 3AM Debug Scenarios

1. OOMKilled Errors: Memory limits exceeded (3GB on SMALL instances)
2. Network Timeouts: Retry logic missing for dependency downloads
3. Dependency Conflicts: Unpinned versions causing random failures
4. YAML Syntax Errors: Buildspec indentation breaking builds

Cache Failure Patterns

• Cache Keys Changing: Timestamps/git hashes in paths prevent cache hits
• Wrong Directory Caching: Mismatched paths between cache config and actual dependencies
• Cross-Region Cache Issues: Cache stored in different region than build
• Permissions Problems: IAM roles lacking S3 cache bucket access

Resource Requirements & Thresholds

Memory Limits by Instance Type

• BUILD_GENERAL1_SMALL: 3GB RAM (fails with large test suites)
• BUILD_GENERAL1_MEDIUM: 7GB RAM (handles most webpack builds)
• BUILD_GENERAL1_LARGE: 15GB RAM (overkill for most applications)

Performance Thresholds

• Optimization Worthy: Builds over 10 minutes or $100/month cost
• Skip Optimization: Builds under 3 minutes or storage under $50/month
• Parallelization Break-even: Independent test suites with 5+ minute runtime

Cost Monitoring Targets

• Cost per build: Under $0.50
• Cost per deployment: Under $2.00
• Monthly cost trend: Flat or declining
• Alert threshold: 20% week-over-week increase

Implementation Decision Tree

When to Use Parallel Builds

Use When:

• Independent test suites (frontend/backend)
• Multiple environment builds
• Large codebases with isolated modules

Avoid When:

• Dependencies between build steps
• Small builds under 5 minutes
• Hitting CodeBuild concurrent limits (100 builds)

Caching Strategy Selection

S3 Caching: Dependencies over 50MB, cross-build persistence needed
Local Caching: Repeated builds, same region, can handle random cache loss
No Caching: Small dependencies under 20MB, first-time builds

Instance Size Selection Criteria

1. Monitor current CPU/memory usage for 30 days
2. CPU under 50% consistently → downsize
3. Memory over 80% → upsize
4. Don't guess based on application type

Breaking Points & Failure Scenarios

UI Performance Degradation

• Breaking Point: 1000+ spans in distributed tracing makes debugging impossible
• Cache Eviction: Local cache randomly nukes, breaking dependency on it
• Build Timeouts: Network latency adds 5-10 minutes to fresh dependency downloads
• Spot Instance Termination: Mid-build interruption requires graceful handling

Cost Explosion Triggers

• Artifact Accumulation: 500MB Docker images forgotten in S3 for months
• Cross-Region Transfers: Automatic replication without considering costs
• Parallel Build Abuse: 10 concurrent 2-minute builds cost same as 1 20-minute build
• Large Instance Defaults: Teams choosing LARGE instances without measurement

Operational Warnings

What Official Documentation Doesn't Tell You

• S3 Caching Latency: Can be slower than fresh downloads for small files
• Local Cache Reliability: Gets randomly evicted, cannot depend on it
• Custom Image Maintenance: Security updates required monthly or vulnerability risk
• Parallel Resource Contention: 4 jobs on 2-CPU instance slower than sequential

Production vs Development Reality

• Default Settings Fail: CodeBuild defaults optimized for simplicity, not production
• Resource Limits Hidden: 3GB memory limit not obvious until OOMKilled
• Cross-Region Costs: Data transfer charges invisible until bill arrives
• Cache Dependencies: Build correctness must not depend on cache availability

Automation Requirements

Essential Cleanup Automation

# Delete artifacts older than 30 days
cutoff_date = datetime.now() - timedelta(days=30)
# CloudWatch logs: 14 days debug, 90 days audit
# S3 lifecycle policies: Automatic transition to cheaper storage

Monitoring That Prevents Issues

• Cost Anomaly Detection: 20% week-over-week alerts
• Build Duration Tracking: Regression detection for performance
• Cache Hit Rate Monitoring: Alerts when caching stops working
• Resource Utilization: Automatic right-sizing recommendations

Success Metrics & Validation

Before/After Measurement Protocol

1. Baseline Period: 30 days current performance and costs
2. Test Implementation: Apply to non-critical builds first
3. Validation Period: 14 days measurement post-optimization
4. Rollout Decision: Based on measurable improvement

Key Performance Indicators

• Build Speed: Average duration trending down
• Cost Efficiency: Cost per build decreasing
• Reliability: Failure rate not increasing from optimizations
• Developer Experience: Faster feedback cycles, higher satisfaction

This reference enables automated decision-making for AWS Developer Tools optimization while preserving critical operational intelligence about failure modes, hidden costs, and real-world implementation challenges.