AWS AI/ML Troubleshooting - What Actually Breaks in Production

Today is Friday, September 05, 2025. I've been debugging AWS AI disasters for three years now, usually at 2am when everything's on fire. AWS error messages are designed by sadists. "UnexpectedStatusException" tells you fuck-all. "InternalServerError" could be a typo in your JSON or AWS having a bad day - who knows?

The Five Production Nightmares That Will Ruin Your Weekend

1. SageMaker Training Jobs That Die Mysteriously

The Scenario

Your model trains fine for hours, then dies with "UnexpectedStatusException" at like 85% completion. Logs are empty, CloudWatch looks normal, and you just burned through a bunch of GPU credits for nothing.

Here's the thing - SageMaker training jobs run in isolated Docker containers with S3 access. When they fail mysteriously, the container just dies without telling you why. I've seen this shit dozens of times. The official SageMaker troubleshooting guide barely mentions this stuff.

What Actually Happens

Usually it's one of these (but AWS won't tell you which):

• Out of memory during checkpointing - Your model grew larger than expected and can't save checkpoints (memory management guide)
• S3 permissions changed mid-training - Someone modified IAM policies while your job was running (IAM troubleshooting)
• Spot instance termination - AWS needed your capacity back and gave you 2 minutes warning (managed spot training docs)
• Docker container timeout - Your training script hit an infinite loop or deadlock (container troubleshooting)

Debug Strategy That Works

1. Check CloudWatch Logs immediately - Go to /aws/sagemaker/TrainingJobs/[your-job-name] (CloudWatch logs guide)
2. Look for the real error - Search for "ERROR", "Exception", "killed", or "terminated" (log analysis patterns)
3. Check memory usage - Use nvidia-smi output in logs or CloudWatch container insights (container insights setup)
4. Verify S3 access - Test if your training job can still read/write to S3 buckets (S3 permissions debugging)

## Add this to your training script for better debugging
import logging
import psutil
import subprocess

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

def debug_system_health():
    """Log system health info before each training step"""
    memory = psutil.virtual_memory()
    disk = psutil.disk_usage('/')
    
    logger.info(f"Memory: {memory.percent}% used, {memory.available / (1024**3):.1f}GB available")
    logger.info(f"Disk: {disk.percent}% used, {disk.free / (1024**3):.1f}GB free")
    
    # GPU memory if available
    try:
        result = subprocess.run(['nvidia-smi', '--query-gpu=memory.used,memory.total', 
                               '--format=csv,noheader,nounits'], 
                               capture_output=True, text=True)
        logger.info(f"GPU Memory: {result.stdout.strip()}")
    except:
        pass

## Call this every few training steps
if step % 100 == 0:
    debug_system_health()

Pro tip

Enable SageMaker Debugger rules to catch memory issues automatically. Costs extra but saves your sanity when things break at 2am.

2. Bedrock Rate Limiting That Kills Your Demo

The Scenario

Your app works perfectly in testing. During the board presentation, every request returns "ThrottlingException: Rate exceeded" and you're standing there like an idiot explaining why your AI can't write a simple email.

What Actually Happens

Bedrock quotas are garbage. Claude 3.5 Sonnet has way lower limits in some regions - like 100k tokens/min instead of the 400k you get in us-east-1. Nova Pro quotas change randomly and are different everywhere. Your demo probably uses 3x more tokens than you tested with because of retries and error handling.

I learned this the hard way when our board demo hit rate limits. Turns out we tested in us-west-2 but somehow the demo was routing through eu-central-1 which has way shittier quotas.

Emergency Fixes

import time
import random
from botocore.exceptions import ClientError

def bedrock_with_backoff(bedrock_client, **kwargs):
    """Implement exponential backoff for Bedrock calls"""
    max_retries = 5
    base_delay = 1
    
    for attempt in range(max_retries):
        try:
            return bedrock_client.invoke_model(**kwargs)
        except ClientError as e:
            if e.response['Error']['Code'] == 'ThrottlingException':
                if attempt == max_retries - 1:
                    raise
                
                # Exponential backoff with jitter
                delay = (base_delay * 2 ** attempt) + random.uniform(0, 1)
                print(f"Rate limited, waiting {delay:.1f}s before retry {attempt + 1}")
                time.sleep(delay)
            else:
                raise

Long-term Fix

Request quota increases 48 hours before important demos. AWS takes 1-3 business days because bureaucracy. Set up billing alerts so you don't accidentally spend the budget on retries.

3. Model Inference Endpoints That Randomly Timeout

The Scenario

Your endpoint works fine for weeks, then suddenly starts timing out on 30% of requests. Users are complaining, metrics show the model is still running, but responses just... stop coming back.

Root Causes I've Actually Seen

• Model memory leaks - Inference containers slowly eat memory until they crash
• Container health check failures - Load balancer marks healthy instances as unhealthy
• Cold start cascades - Auto-scaling spins up new instances that take 5 minutes to initialize
• Input validation hanging - Malformed requests cause the model to hang indefinitely

Debugging Approach

1. Check endpoint CloudWatch metrics first:

   • ModelLatency - Are response times increasing?
   • Invocation4XXErrors - Client-side issues
   • Invocation5XXErrors - Server-side problems
   • CPUUtilization and MemoryUtilization - Resource exhaustion

2. Examine container logs in CloudWatch:

   aws logs filter-log-events \
     --log-group-name /aws/sagemaker/Endpoints/your-endpoint-name \
     --start-time 1693900800000 \
     --filter-pattern "ERROR"
   

3. Test endpoint directly to isolate the problem:

   import boto3
   import json
   
   runtime = boto3.client('sagemaker-runtime')
   
   # Test with minimal payload
   response = runtime.invoke_endpoint(
       EndpointName='your-endpoint',
       ContentType='application/json',
       Body=json.dumps({"text": "test"})
   )
   

Quick Fixes

• Restart the endpoint - Sometimes containers just get weird
• Scale up instance count - More instances = better fault tolerance
• Switch to a larger instance type - May be hitting memory/CPU limits

4. Cross-Region Deployment Hell

The Scenario

Your model works perfectly in us-east-1. Deploy the exact same code to eu-central-1 and everything breaks with region-specific errors that make no sense.

What Breaks Across Regions

• Model availability - Not all Bedrock models work in all regions
• IAM role ARNs - Hardcoded role references fail in new regions
• S3 bucket permissions - Cross-region access denied errors
• VPC configurations - Subnets and security groups don't exist in new region
• KMS keys - Customer-managed keys are region-specific

Multi-Region Debug Checklist

## 1. Verify model availability
aws bedrock list-foundation-models --region eu-central-1 \
  --query 'modelSummaries[?contains(modelId, `claude`)]'

## 2. Check IAM role exists in target region
aws iam get-role --role-name YourSageMakerRole --region eu-central-1

## 3. Test S3 access from target region
aws s3 ls s3://your-model-bucket --region eu-central-1

## 4. Verify VPC resources
aws ec2 describe-subnets --region eu-central-1 --filters "Name=tag:Name,Values=ml-*"

Architecture Fix

Use CloudFormation StackSets to deploy the same shit in multiple regions. Copy-paste infrastructure is error-prone and will bite you during an outage.

5. Nova Model Cold Starts Killing User Experience

The Scenario

Users click "Analyze Document" and wait 30 seconds staring at a loading spinner because your Nova model was idle for 20 minutes and needs to wake up.

Bedrock models sit behind load balancers and auto-scale based on demand. When they've been idle - and I think it's like 15-20 minutes but could be less - the first request after that triggers a cold start while AWS spins up inference capacity. It's annoying as hell.

Cold Start Reality

• First request after idle: maybe 8-15 seconds for Nova Pro, sometimes longer
• Complex requests: 20+ seconds if you're unlucky
• User patience: 3 seconds before they think it's broken and start clicking refresh

Mitigation Strategies

1. Keep-Alive Pinging:

import schedule
import time
import boto3

bedrock = boto3.client('bedrock-runtime')

def keep_warm():
    """Send minimal request to prevent cold starts"""
    try:
        bedrock.invoke_model(
            modelId='amazon.nova-pro-v1:0',
            body=json.dumps({
                "anthropic_version": "bedrock-2023-05-31",
                "max_tokens": 10,
                "messages": [{"role": "user", "content": "ping"}]
            })
        )
        print("Keep-alive successful")
    except Exception as e:
        print(f"Keep-alive failed: {e}")

## Ping every 5 minutes during business hours
schedule.every(5).minutes.do(keep_warm)

2. Async Processing with Status Updates:

## Don't make users wait for long-running requests
def process_document_async(document_id):
    # Return immediately with job ID
    job_id = str(uuid.uuid4())
    
    # Process in background
    threading.Thread(
        target=analyze_document_background,
        args=(job_id, document_id)
    ).start()
    
    return {"job_id": job_id, "status": "processing"}

def check_job_status(job_id):
    # Let users poll for results
    return {"status": "completed", "results": "..."}

The Debug Toolchain That Actually Works

Essential CloudWatch Queries That Actually Work

Find SageMaker Training Failures

fields @timestamp, @message
| filter @message like /ERROR/
| sort @timestamp desc
| limit 50

Track Bedrock Rate Limiting

fields @timestamp, @message  
| filter @message like /ThrottlingException/
| stats count() by bin(5m)

Memory Issues

fields @timestamp, @message
| filter @message like /OutOfMemoryError/ or @message like /killed/
| sort @timestamp desc

Custom Monitoring That Actually Prevents Disasters

Beyond basic CloudWatch metrics, you need custom tracking for AI-specific failures:

import boto3
import time

def setup_ai_monitoring():
    """Set up custom CloudWatch metrics for AI workloads"""
    cloudwatch = boto3.client('cloudwatch')
    
    # Track token usage per hour
    cloudwatch.put_metric_data(
        Namespace='AI/Production',
        MetricData=[
            {
                'MetricName': 'TokensPerHour',
                'Value': token_count,
                'Unit': 'Count',
                'Dimensions': [
                    {'Name': 'Model', 'Value': 'nova-pro'},
                    {'Name': 'Application', 'Value': 'document-analysis'}
                ]
            }
        ]
    )
    
    # Track cold start frequency
    cloudwatch.put_metric_data(
        Namespace='AI/Performance',
        MetricData=[
            {
                'MetricName': 'ColdStarts',
                'Value': 1 if response_time > 10 else 0,
                'Unit': 'Count'
            }
        ]
    )

Emergency Response Playbook

When Everything is Broken

1. Check AWS Service Health first - don't waste time debugging if AWS is down
2. Switch regions if possible
3. Enable debug logging
4. Scale up resources manually
5. Rollback to last known good state

When Models Stop Working

1. Compare recent logs with working periods
2. Test with minimal examples - find the breaking change
3. Check for silent AWS updates - models change without warning
4. Validate input data format - API changes break everything

Shit I've Learned the Hard Way

Cost alerts are not optional

Had hyperparameter tuning jobs fail and restart in a loop over a long weekend once. We burned through like 30-something thousand dollars before anyone noticed on Monday morning. Could've been 40K, honestly not sure exactly. Now I set billing alerts for anything over $500/day and actually check them.

Demo on a different region

Did a client demo that worked perfectly in testing, then hit rate limits during the actual presentation. Turns out we tested in us-west-2 but somehow the demo was routing through eu-central-1 which has garbage quotas. Still have no clue how that routing happened - maybe a DNS thing? Always check which region you're actually hitting, not just which one you think you configured.

Models change without warning

Production model outputs shifted after what turned out to be an undocumented AWS model update. Took us weeks to figure out why our accuracy dropped. Started running daily validation tests after that mess, though they don't catch everything.

These aren't theoretical problems - they happen to everyone. The difference between teams that survive and teams that burn out is having debugging strategies that actually work when everything is on fire.

Advanced Debugging - Beyond the Basic Shit

After fixing AWS AI disasters that cost real money, here's the debugging stuff that actually works when CloudWatch logs give you nothing useful.

X-Ray Tracing for Multi-Service AI Workflows

The Problem: Your AI workflow involves Lambda → Bedrock → SageMaker → DynamoDB. When it breaks, good luck figuring out which service is the culprit.

X-Ray creates a service map showing request flow between all these services. Each service appears as a node with latency and error rate data. It's actually pretty useful when everything goes to shit. Check the X-Ray developer guide for setup details.

The Solution: AWS X-Ray traces requests across services and shows you exactly where things break. I use this all the time now. The X-Ray SDK documentation shows how to instrument your code, and the service map guide explains how to read the output. For advanced debugging, check out X-Ray trace analysis and performance tuning patterns.

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
import boto3

## Patch all AWS SDK calls for automatic tracing
patch_all()

@xray_recorder.capture('process_document')
def process_document(document_id):
    with xray_recorder.in_subsegment('bedrock_analysis'):
        bedrock = boto3.client('bedrock-runtime')
        # Bedrock calls are automatically traced
        response = bedrock.invoke_model(...)
    
    with xray_recorder.in_subsegment('sagemaker_inference'):
        sagemaker = boto3.client('sagemaker-runtime')
        # This will show up as a separate trace segment
        result = sagemaker.invoke_endpoint(...)
    
    return result

What This Shows You:

• Exact latency breakdown across services
• Which service threw the first error
• Network timeouts vs application errors
• Cold start detection across the workflow

Real War Story: Client's document processing was randomly failing. CloudWatch showed everything looked fine, but X-Ray revealed that like 15% of Bedrock calls were timing out after exactly 30 seconds. Took us two fucking days to figure out it was a VPC network config issue. Would've been impossible to find without tracing, honestly. The Lambda debugging guide and Lambda best practices would have helped if we'd found them sooner.

Custom CloudWatch Insights Queries That Actually Help

Standard CloudWatch queries are useless for AI debugging. Here are the queries I actually use. For comprehensive logging strategies, check out Lambda logging best practices and the serverless debugging guide:

Find Memory Issues Before They Kill Training Jobs:

fields @timestamp, @message
| filter @message like /memory/
| stats count() by bin(5m), @message
| sort @timestamp desc

Track Token Usage Patterns for Cost Optimization:

fields @timestamp, @message
| parse @message "tokens: *" as token_count
| stats avg(token_count), max(token_count), sum(token_count) by bin(1h)
| sort @timestamp desc

Detect Model Performance Degradation:

fields @timestamp, @message  
| parse @message "confidence: *" as confidence_score
| stats avg(confidence_score) by bin(30m)
| sort @timestamp desc

Identify Rate Limiting Patterns:

fields @timestamp, @message
| filter @message like /ThrottlingException/ or @message like /429/
| stats count() by bin(1m)
| sort @timestamp desc

Building Custom Monitoring for AI-Specific Metrics

CloudWatch's default metrics miss the stuff that actually matters for AI workloads.

Here's what I track:

import boto3
import json
from datetime import datetime

class AIMetricsCollector:
    def __init__(self):
        self.cloudwatch = boto3.client('cloudwatch')
    
    def track_model_performance(self, model_name, accuracy, latency, cost):
        """Track metrics that correlate with business impact"""
        metrics = [
            {
                'MetricName': 'ModelAccuracy',
                'Value': accuracy,
                'Unit': 'Percent',
                'Dimensions': [{'Name': 'Model', 'Value': model_name}]
            },
            {
                'MetricName': 'ResponseLatency',
                'Value': latency,
                'Unit': 'Milliseconds',
                'Dimensions': [{'Name': 'Model', 'Value': model_name}]
            },
            {
                'MetricName': 'CostPerRequest',
                'Value': cost,
                'Unit': 'None',
                'Dimensions': [{'Name': 'Model', 'Value': model_name}]
            }
        ]
        
        self.cloudwatch.put_metric_data(
            Namespace='AI/Production',
            MetricData=metrics
        )
    
    def detect_data_drift(self, input_data, training_baseline):
        """Monitor for input data changes that affect model performance"""
        # Simple statistical comparison
        drift_score = calculate_distribution_shift(input_data, training_baseline)
        
        self.cloudwatch.put_metric_data(
            Namespace='AI/Quality',
            MetricData=[{
                'MetricName': 'DataDrift',
                'Value': drift_score,
                'Unit': 'None'
            }]
        )
        
        # Alert if drift exceeds threshold
        if drift_score > 0.3:
            self.send_alert(f"Data drift detected: {drift_score:.2f}")

def calculate_distribution_shift(current_data, baseline_data):
    """Calculate KL divergence or similar metric"""
    # Implementation depends on your data type
    pass

The Debug Incident Response Playbook

When AWS AI services break in production, you have maybe 15 minutes before users start complaining and executives start asking questions. Here's the approach I use when everything's on fire:

Immediate Triage (First 5 minutes)

1. Check AWS Service Health - don't waste time if AWS is having issues
2. Test basic connectivity to your services
3. Check recent deployments in CloudTrail
4. Verify service quotas aren't blocking you

Phase 2: Service-Specific Debugging (5-15 minutes)

#!/bin/bash
## Emergency AWS AI debug script

echo "=== SageMaker Status ==="
aws sagemaker list-training-jobs --status-equals InProgress --max-results 10
aws sagemaker list-endpoints --status-equals InService --max-results 10

echo "=== Recent Bedrock Errors ==="
aws logs filter-log-events \
  --log-group-name "/aws/lambda/your-function" \
  --start-time $(date -d '1 hour ago' +%s)000 \
  --filter-pattern "ThrottlingException"

echo "=== CloudWatch Alarms ==="
aws cloudwatch describe-alarms --state-value ALARM --max-records 10

echo "=== Recent IAM Changes ==="
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutRolePolicy \
  --start-time $(date -d '4 hours ago' --iso-8601) \
  --max-items 5

Deep Debugging (15+ minutes)

• Enable detailed logging and X-Ray tracing
• Capture VPC Flow Logs for network issues
• Compare with known-good baseline metrics
• Test with minimal reproduction cases

Performance Optimization Through Debugging Data

Identifying Bottlenecks in AI Workflows:

Most performance issues in AWS AI come from these patterns:

• Sequential processing when you could parallelize
• Synchronous waits for long-running operations
• Cold starts from unused services
• Oversized models for simple tasks

import asyncio
import time
from concurrent.futures import ThreadPoolExecutor

class OptimizedAIWorkflow:
    def __init__(self):
        self.bedrock = boto3.client('bedrock-runtime')
        self.sagemaker = boto3.client('sagemaker-runtime')
    
    async def process_documents_parallel(self, documents):
        """Process multiple documents concurrently"""
        tasks = []
        for doc in documents:
            task = asyncio.create_task(self.process_single_document(doc))
            tasks.append(task)
        
        results = await asyncio.gather(*tasks, return_exceptions=True)
        return results
    
    async def process_single_document(self, document):
        """Process one document with performance tracking"""
        start_time = time.time()
        
        try:
            # Use appropriate model size for task complexity
            model_id = self.select_optimal_model(document)
            
            result = await self.call_bedrock_async(model_id, document)
            
            processing_time = time.time() - start_time
            self.log_performance_metric(model_id, processing_time, len(document))
            
            return result
            
        except Exception as e:
            self.log_error_with_context(document, e)
            raise
    
    def select_optimal_model(self, document):
        """Choose cheapest model that can handle the task"""
        word_count = len(document.split())
        
        if word_count < 1000:
            return "amazon.nova-micro-v1:0"  # Cheapest for simple stuff
        elif word_count < 5000:
            return "amazon.nova-lite-v1:0"   # Usually good enough
        else:
            return "amazon.nova-pro-v1:0"    # When you need the good shit

Advanced Error Pattern Recognition

Intermittent Failures: These are the worst because they're hard to reproduce. Use these techniques:

class ErrorPatternDetector:
    def __init__(self):
        self.error_history = []
        self.cloudwatch = boto3.client('cloudwatch')
    
    def log_request_outcome(self, request_id, success, error_type=None, 
                          timestamp=None, context=None):
        """Log every request for pattern analysis"""
        self.error_history.append({
            'request_id': request_id,
            'success': success,
            'error_type': error_type,
            'timestamp': timestamp or datetime.utcnow(),
            'context': context  # user_id, document_type, etc.
        })
        
        # Analyze patterns every 100 requests
        if len(self.error_history) % 100 == 0:
            self.analyze_error_patterns()
    
    def analyze_error_patterns(self):
        """Detect patterns in failures"""
        recent_errors = [e for e in self.error_history[-100:] if not e['success']]
        
        if len(recent_errors) > 10:  # roughly 10% error rate, adjust as needed
            patterns = self.find_patterns(recent_errors)
            if patterns:
                self.alert_ops_team(f"Error pattern detected: {patterns}")
    
    def find_patterns(self, errors):
        """Look for common characteristics in errors"""
        patterns = []
        
        # Time-based patterns
        error_hours = [e['timestamp'].hour for e in errors]
        if len(set(error_hours)) < 3:  # Errors clustered in time
            patterns.append(f"Errors concentrated in hours: {set(error_hours)}")
        
        # User-based patterns  
        error_users = [e['context'].get('user_id') for e in errors if e['context']]
        user_counts = {user: error_users.count(user) for user in set(error_users)}
        if max(user_counts.values()) > 5:  # One user causing many errors
            patterns.append(f"High error user: {max(user_counts, key=user_counts.get)}")
        
        # Error type clustering
        error_types = [e['error_type'] for e in errors]
        if len(set(error_types)) == 1:  # All same error type
            patterns.append(f"Single error type: {error_types[0]}")
        
        return patterns

Cost-Optimized Debugging Infrastructure

The Problem: Debug logging and monitoring can double your AWS bill if you're not careful. I learned this the hard way.

Smart Logging Strategy I Use Now:

import logging
import os
from datetime import datetime, timedelta

class CostOptimizedLogger:
    def __init__(self, max_daily_cost=10.00):  # $10/day logging budget
        self.max_daily_cost = max_daily_cost
        self.current_cost = 0
        self.daily_log_volume = 0  # bytes
        self.setup_logging()
    
    def setup_logging(self):
        """Configure logging with cost controls"""
        # Sample logs during high-cost periods
        if self.current_cost > self.max_daily_cost * 0.8:
            log_level = logging.WARNING  # Reduce verbosity
            sample_rate = 0.1  # Log only 10% of events
        else:
            log_level = logging.INFO
            sample_rate = 1.0
        
        logging.basicConfig(level=log_level)
        self.sample_rate = sample_rate
    
    def smart_log(self, level, message, force=False):
        """Log with cost awareness"""
        if not force and random.random() > self.sample_rate:
            return
        
        # Estimate logging cost (roughly $0.50 per GB ingested)
        message_size = len(message.encode('utf-8'))
        estimated_cost = (message_size / (1024**3)) * 0.50
        
        if self.current_cost + estimated_cost > self.max_daily_cost:
            if level >= logging.ERROR:  # Always log errors
                logging.log(level, f"[COST_LIMITED] {message}")
            return
        
        self.current_cost += estimated_cost
        self.daily_log_volume += message_size
        logging.log(level, message)

Automated Root Cause Analysis

Build systems that debug themselves:

class AISystemHealthChecker:
    def __init__(self):
        self.known_patterns = self.load_issue_patterns()
        self.cloudwatch = boto3.client('cloudwatch')
    
    def diagnose_system_health(self):
        """Automatically identify likely root causes"""
        symptoms = self.collect_symptoms()
        diagnosis = self.match_patterns(symptoms)
        
        if diagnosis:
            self.suggest_fixes(diagnosis)
        
        return diagnosis
    
    def collect_symptoms(self):
        """Gather metrics that indicate problems"""
        symptoms = {}
        
        # API response times
        symptoms['high_latency'] = self.check_metric(
            'AWS/Bedrock', 'ResponseLatency', threshold=10000  # 10 seconds
        )
        
        # Error rates
        symptoms['high_errors'] = self.check_metric(
            'AWS/SageMaker', 'Invocation4XXErrors', threshold=0.05  # 5%
        )
        
        # Cost anomalies  
        symptoms['cost_spike'] = self.check_cost_anomaly()
        
        # Resource utilization
        symptoms['memory_pressure'] = self.check_metric(
            'AWS/SageMaker', 'MemoryUtilization', threshold=0.9  # 90%
        )
        
        return {k: v for k, v in symptoms.items() if v}
    
    def match_patterns(self, symptoms):
        """Match symptoms to known issue patterns"""
        for pattern in self.known_patterns:
            if all(symptom in symptoms for symptom in pattern['symptoms']):
                return pattern
        
        return None
    
    def suggest_fixes(self, diagnosis):
        """Provide actionable fixes based on diagnosis"""
        fixes = diagnosis.get('fixes', [])
        
        for fix in fixes:
            if fix['type'] == 'scale_up':
                self.auto_scale_resources(fix['resource'])
            elif fix['type'] == 'restart':
                self.restart_service(fix['service'])
            elif fix['type'] == 'alert':
                self.notify_team(fix['message'])
    
    def load_issue_patterns(self):
        """Load known patterns from previous incidents"""
        return [
            {
                'name': 'memory_exhaustion',
                'symptoms': ['high_latency', 'memory_pressure'],
                'fixes': [
                    {'type': 'scale_up', 'resource': 'endpoint_instances'},
                    {'type': 'alert', 'message': 'Memory exhaustion detected'}
                ]
            },
            {
                'name': 'rate_limiting',
                'symptoms': ['high_errors', 'high_latency'],
                'fixes': [
                    {'type': 'alert', 'message': 'Request quota increase needed'},
                    {'type': 'enable', 'feature': 'exponential_backoff'}
                ]
            }
        ]

This level of debugging infrastructure separates production-ready systems from hobby projects. When AWS AI services break - and trust me, they will - having systematic debugging approaches saves both your sanity and your company's reputation. I wish I'd had this stuff set up three years ago when I was debugging everything manually like an idiot.