AWS AI/ML Infrastructure: Production-Ready Implementation Guide

Service Selection Decision Matrix

Amazon Bedrock vs SageMaker vs Custom Infrastructure

Criteria	Amazon Bedrock	Amazon SageMaker	Custom EC2/EKS
Cost Structure	$0.01-0.10 per 1K tokens	$50-500/month per endpoint (idle)	EC2 costs + engineering overhead
Scaling Behavior	Automatic (rate limits at scale)	5+ minute cold start delays	Manual implementation required
Time to Production	2-3 days	2-4 weeks	2-6 months
Use When	<1M requests/month, API-based models	Custom models, >1M requests/month	Compliance requirements, specialized needs
Performance Ceiling	Rate throttling during peak hours	Predictable until auto-scaling triggers	Depends on implementation quality
Operational Overhead	Minimal - API calls only	Moderate - instance management	Maximum - everything custom

Decision Thresholds

• Under 100K requests/month: Use Bedrock
• 100K-1M requests/month: Evaluate both based on token vs endpoint costs
• Over 1M requests/month: SageMaker likely more cost-effective
• Sub-100ms latency required: SageMaker with warm instances
• Custom model requirements: SageMaker mandatory

Critical Production Failure Modes

Auto-Scaling Limitations

• Problem: GPU instances require 5+ minutes to initialize
• Impact: 50% of users experience timeouts during traffic spikes
• Mitigation: Maintain warm instance pools, implement predictive scaling
• Cost: 24/7 warm instances increase costs by 200-300%

Multi-Model Resource Wars

• Problem: Models competing for memory on shared endpoints
• Symptoms: Random empty responses, ResourceLimitExceeded errors
• Solution: Separate endpoints for critical models
• Investigation Time: 3+ days to identify root cause

Training Job Interruptions

• Spot Instance Failure Rate: 30-50% for jobs >8 hours
• Cost Savings: 50-70% with spot instances
• Recovery Requirement: Checkpoint every 30 minutes minimum
• Lost Work: 90% complete jobs can be terminated without warning

Cost Control Configuration

Budget Protection (Mandatory)

Daily Alert: $500
Weekly Alert: $2000
Monthly Alert: $5000
Instance Approval: Anything > ml.g4dn.xlarge

GPU Instance Costs (Per Hour)

• ml.g4dn.xlarge: $1.20/hour
• ml.p3.2xlarge: $3.06/hour
• ml.p4d.24xlarge: $32.77/hour
• Weekend Cost: p4d.24xlarge = $1,574 for 48 hours

Cost Optimization Strategies

1. Spot Instances: 50-70% savings, handle interruptions
2. Batch Processing: Avoid real-time endpoints when possible
3. Aggressive Caching: Cache inference results for repeated queries
4. Instance Shutdown: Automatic termination for idle resources

Security Architecture Requirements

Network Isolation (Non-Negotiable)

• VPC endpoints mandatory for all ML traffic
• No public internet access for ML workloads
• Security audit failure guaranteed without proper isolation

Encryption Standards

• All data encrypted in transit and at rest using KMS
• Training data, models, and inference requests must be encrypted
• Performance impact: 5-10% latency increase

IAM Permission Complexity

• ML workflows require access to: S3, ECR, CloudWatch, SageMaker, Bedrock
• Initial setup: Start with broad permissions
• Production lockdown: Plan 1 week for permission debugging
• Common error: AccessDenied with unclear root cause

Monitoring and Alerting Configuration

Traditional Monitoring Limitations

• HTTP 200 OK responses while serving incorrect predictions
• Infrastructure metrics don't indicate model performance degradation
• CPU/memory utilization irrelevant for model accuracy

Essential ML Metrics

1. Model Accuracy: Compare predictions vs ground truth when available
2. Prediction Confidence: Low confidence scores indicate potential issues
3. Data Drift Detection: Input distribution changes break models
4. Business Impact: Conversion rates, user satisfaction metrics

Alert Thresholds

• Model accuracy drop >10% from baseline
• Prediction confidence <70% for >5% of requests
• Error rate >1% for inference endpoints
• Response time >2 seconds for real-time endpoints

Production Deployment Patterns

Rollback Strategy (Required Before Deployment)

1. Blue-Green: 100% traffic switch, expensive (2x resources)
2. Canary: 5% traffic to new model, gradual increase
3. A/B Testing: Split traffic between model versions
4. Circuit Breaker: Automatic rollback on error threshold

Deployment Failure Recovery

• Rollback immediately, debug offline
• Cache invalidation during rollback causes additional failures
• Users receiving bad predictions worse than no predictions
• Model versioning must sync across regions

Infrastructure Scaling Thresholds

Performance Bottlenecks

• 1000+ concurrent requests: Standard endpoints become unreliable
• 100GB+ training data: Local processing fails, streaming required
• Multi-region deployment: Model artifact sync complexity increases exponentially

Capacity Planning

• Budget 3x expected load for auto-scaling delays
• Plan for 10-minute warmup time during traffic spikes
• Cache hit ratio must exceed 80% for cost-effective operation

Implementation Timeline (Realistic)

Phase 1: Foundation (Weeks 1-4)

• Billing protection and cost alerts
• VPC and security configuration
• Bedrock proof-of-concept
• Failure Point: Skipping security setup leads to 6-month refactoring

Phase 2: Custom Models (Weeks 5-16)

• SageMaker environment setup
• Training infrastructure with checkpointing
• Initial model deployment
• Failure Point: Underestimating IAM complexity adds 2-4 weeks

Phase 3: Production (Weeks 17-24)

• Monitoring and alerting implementation
• Deployment automation
• Load testing and optimization
• Failure Point: No rollback plan causes extended outages

Phase 4: Optimization (Month 6+)

• Cost optimization implementation
• Performance tuning
• Multi-model architecture
• Ongoing: 50% efficiency improvement over 6 months typical

Common Implementation Failures

Over-Engineering (60% of Projects)

• Building custom infrastructure when Bedrock APIs sufficient
• Implementing distributed training for single-GPU workloads
• Creating complex MLOps pipelines for prototype models

Cost Explosion (40% of Projects)

• No billing alerts during experimentation
• Leaving training clusters running during non-business hours
• Under-provisioning leading to emergency scaling at premium rates

Security Retrofitting (30% of Projects)

• Implementing VPC isolation after production deployment
• Adding encryption to existing data pipelines
• IAM policy restructuring in production environment

Resource Requirements by Use Case

API-Based Applications (Bedrock)

• Engineering Time: 1-2 weeks
• Expertise Required: API integration, prompt engineering
• Ongoing Costs: Token-based, scales with usage
• Maintenance: Minimal, AWS-managed

Custom Model Development (SageMaker)

• Engineering Time: 2-6 months
• Expertise Required: ML engineering, DevOps, monitoring
• Ongoing Costs: Fixed endpoint costs + usage
• Maintenance: High, requires ongoing optimization

Enterprise ML Platform (Custom)

• Engineering Time: 6-18 months
• Expertise Required: Distributed systems, Kubernetes, ML infrastructure
• Ongoing Costs: Infrastructure + dedicated team
• Maintenance: Maximum, full platform responsibility