ML Model Production Deployment - AI-Optimized Technical Reference

Critical Failure Patterns

Docker Deployment Failures

• Architecture mismatch: ARM vs x86 causes "exec user process caused: exec format error"
• Dependency conflicts: numpy.distutils deprecated in Python 3.12 conflicts with scikit-learn 1.2.2
• Memory requirements: Models requiring 6GB RAM fail on 4GB production instances
• Base image selection: ubuntu:latest (1GB) vs python:3.11-slim significantly impacts deployment speed
• Requirements management: pip freeze creates broken dependencies from mixed environments

Kubernetes Resource Management

• OOMKilled errors: Pods crash without proper memory limits
• Pending state: Pods stuck due to resource constraints or node availability
• HPA scaling disasters: Can spin up 58 GPU instances at $3.06/hour each ($4,200 bill example)
• Health check failures: Load balancers route to dead pods when checks only verify process existence

Configuration Requirements

Docker Production Settings

FROM python:3.11-slim  # Not ubuntu:latest (1GB overhead)
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
EXPOSE 8000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

Multi-stage builds: Reduce 2GB+ images, save hours of deployment time
Layer optimization: Dependencies first (change least), code last (changes most)
Required files: .dockerignore prevents .git history inclusion

Kubernetes Resource Limits

resources:
  limits:
    memory: "4Gi"  # Set or nodes die from OOM
    cpu: "2"
  requests:
    memory: "2Gi"
    cpu: "1"

Health Checks That Work

livenessProbe:
  httpGet:
    path: /health/live
    port: 8000
  initialDelaySeconds: 30
  periodSeconds: 10
readinessProbe:
  httpGet:
    path: /health/ready  # Must test model loading, not just process
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 5

Cost Analysis

Cloud Platform Pricing

Service	Cost	Hidden Charges	Breaking Point
AWS SageMaker	$0.065-1.20/hour	Data transfer, storage	4-digit bills without limits
EKS Control Plane	$0.10/hour base	Worker nodes, egress	Always running cost
GPU Instances (p3.2xlarge)	$3.06/hour	CUDA memory sharing conflicts	$8K/month for failed auto-retraining
Data Transfer	$90/TB out of AWS	SSL termination overhead	10,000 predictions/minute

Resource Right-Sizing Strategy

• Start with generous limits, tune down based on actual usage
• Set billing alerts before deployment
• GPU sharing fails when one model consumes all VRAM
• Spot instances: 70% cheaper but disappear before demos

Performance Optimization Trade-offs

Model Optimization Techniques

Method	Speed Gain	Accuracy Impact	Implementation Cost
TensorRT	4x faster	Debugging impossible	CUDA driver dependency
INT8 Quantization	4x speedup	May destroy edge case handling	Post-training: easy but no control
Model Pruning	Variable	Removes model components	Requires retraining
ONNX Conversion	Hardware agnostic	Custom operators unsupported	Format conversion complexity

Critical Warning: INT8 quantization can make models classify 97% of inputs as single category due to softmax rounding

Latency Requirements by Use Case

• Fraud detection: <10ms (transaction timeout)
• Web applications: <100ms (user perception)
• Batch processing: 6+ hours acceptable
• Chatbots: <500ms (user attention)

Monitoring Requirements

Essential Metrics

• System health: Process alive, resource usage
• Model performance: Prediction accuracy, data drift
• Business impact: Conversion rates, revenue metrics
• Infrastructure: GPU utilization, memory leaks

Alert Configuration

• Real problems vs normal variation distinction
• CPU usage thresholds cause false positives
• Model accuracy degradation (95% → 60% over 6 months)
• Automated rollback triggers from traffic variations

Security Implementation

Data Privacy Compliance

• Differential privacy reduces model utility
• Federated learning: complex, expensive implementation
• Basic anonymization insufficient for GDPR
• Budget requirement: Privacy lawyer consultation

Model Security Threats

• Adversarial inputs extract training data
• Input validation catches obvious attacks only
• Sophisticated attacks slip through validation
• Rate limiting stops basic abuse, not botnets

Production Operations

Model Lifecycle Management

• Silent degradation: 95% → 65% accuracy over 6 months unnoticed
• Automated retraining failures: Pipeline crashes, wrong data, poor models deployed
• Version control: Preprocessing pipelines must be versioned with models
• Feature stores: Solve training-serving skew but add complexity and cost

Incident Response Protocol

1. Roll back to previous model version (first step)
2. Check data pipeline before model debugging
3. Distinguish technical metrics from business impact
4. Practice rollbacks during normal operation

Multi-Model Complexity

Ensemble Serving Challenges

• Model A nonsense + Model B timeout = undefined ensemble output
• Load balancer routing affects A/B test validity
• Resource quotas fail during business-critical demands
• Monitoring overhead: separate dashboards per tenant

A/B Testing Reality

• Model performance varies with data difficulty during test periods
• Traffic splitting randomness depends on load balancer behavior
• Statistical significance requires long test periods
• Business metric changes unrelated to model performance

Implementation Timeline Expectations

Realistic Effort Allocation

• 30% model deployment
• 70% maintaining production operations
• 2x initial time estimate, then double again
• Debug time exceeds development time

Common Underestimates

• Environment parity (staging ≠ production)
• Dependency version conflicts
• Resource limit tuning
• Security implementation
• Monitoring setup and alert tuning

Tool Selection Matrix

Requirement	Recommended Tool	Complexity	Cost Impact
Container orchestration	Kubernetes (managed)	High	$0.10/hour base + nodes
Model serving	FastAPI + Docker	Medium	Instance costs only
Monitoring	Prometheus + Grafana	High	Alert fatigue risk
Model registry	MLflow	Medium	Version control essential
GPU inference	NVIDIA Triton	High	Performance critical
Data drift detection	Evidently	Low	Reports ignored until crisis

Critical Success Factors

Build for Debugging, Not Deployment

• Log every request, prediction, error
• Structure logs for grep efficiency
• Distributed tracing (10-20ms overhead acceptable)
• Rollback capability as primary safety mechanism

Operational Intelligence

• Production systems fail gracefully, not perfectly
• Senior engineers have production horror story collections
• Success = survivable systems, not sophisticated infrastructure
• Emergency calls arrive at worst possible times

Resource Optimization Principles

• Manual scaling beats automatic until traffic patterns understood
• Batch processing improves efficiency but adds latency
• Dynamic batching complicates everything
• Right-sizing means running at 80% capacity (no spike room)

Breaking Points and Limits

Known System Limits

• FastAPI: Crashes above 50 concurrent users without scaling
• Kubernetes HPA: Scales up on slow responses, creating more slow pods
• Lambda: 15-second timeout kills long predictions
• TorchServe: Memory leaks require periodic restarts
• Model registries: Version conflicts when preprocessing not tracked

Infrastructure Failure Modes

• SSL certificate expiration breaks service mesh
• Network policies prevent inter-service communication
• Auto-shutdown policies disable dependent databases
• Cost optimization tools shut down during traffic spikes
• Monitoring systems go down without notification

This technical reference provides actionable intelligence for AI systems to make informed decisions about ML deployment strategies, resource allocation, and risk mitigation while preserving all critical operational knowledge from real-world production experience.