Rust Axum Production Deployment - AI-Optimized Technical Reference

Critical Production Failures and Solutions

Environment Failures

• Connection Refused Error: App tries to connect to localhost instead of container service name - use 0.0.0.0 or service name in Docker
• Memory Kill Pattern: Apps mysteriously die with no logs - Kubernetes kills containers exceeding memory limits without clear notification
• Health Check Database Load: 10 load balancer nodes checking every 2 seconds = 50 DB queries/second just for health checks

Docker Networking Failures

• Docker localhost resolution: Doesn't work in containers - must use service names or 0.0.0.0
• Alpine musl issues: Random segfaults with unclear stack traces - use Debian base instead
• Memory limits: 256MB limit will kill apps using 400MB under load with no clear error messages

Resource Requirements and Performance Thresholds

Memory Usage Patterns

• Base usage: 20-50MB idle
• Production minimum: 512MB allocation
• Recommended: 2GB budget (apps consume more than expected under load)
• Breaking point: Containers killed when exceeding limits with minimal logging

Build Time Trade-offs

• Without LTO: 2-minute builds
• With LTO optimization: 8-minute builds, 20% smaller/faster binaries
• Docker multi-stage: Mandatory to avoid 1.5GB images with full Rust toolchain

Database Connection Pool Limits

• Min connections: 2-5
• Max connections: 10-30 (based on database limits)
• Connection timeout failures indicate pool too small or queries too slow
• Monitor connection acquisition times and pool exhaustion

Production Configuration That Actually Works

Dockerfile (Multi-stage, Debian-based)

FROM rust:slim-bookworm AS builder
RUN apt-get update && apt-get install -y pkg-config libssl-dev && rm -rf /var/lib/apt/lists/*

WORKDIR /app
# Layer caching - copy deps first
COPY Cargo.toml Cargo.lock ./
RUN mkdir src && echo "fn main() {}" > src/main.rs
RUN cargo build --release && rm -rf src

COPY src ./src
RUN touch src/main.rs && cargo build --release

FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*
RUN useradd --create-home app
COPY --from=builder /app/target/release/your-app /usr/local/bin/
USER app
EXPOSE 8080
CMD ["your-app"]

Production Cargo.toml Settings

[profile.release]
lto = true          # 20% smaller/faster, 4x longer compile
codegen-units = 1   # Better optimization, slower compile
panic = "abort"     # Smaller binary, no unwinding
strip = true        # Remove debug symbols

Health Check Implementation (Actually Functional)

// Check database connectivity - if DB fails, app is unusable
async fn health_check(State(app_state): State<AppState>) -> Result<Json<serde_json::Value>, StatusCode> {
    match sqlx::query("SELECT 1").execute(&app_state.db_pool).await {
        Ok(_) => Ok(Json(json!({
            "status": "healthy",
            "database": "connected",
            "timestamp": chrono::Utc::now()
        }))),
        Err(e) => {
            tracing::error!("Health check failed: {}", e);
            Err(StatusCode::SERVICE_UNAVAILABLE)
        }
    }
}

// Lightweight readiness check - don't check database here
async fn readiness_check() -> StatusCode {
    StatusCode::OK
}

Platform Deployment Comparison

Platform	Complexity	Monthly Cost	Scaling	Failure Modes
Docker + VPS	Low (Linux knowledge required)	$5-50	Manual (ssh and troubleshoot)	Full responsibility for failures
Kubernetes	Extremely High	$200-1000+	Automatic perfection	Overkill for <10 engineers, complex failure modes
AWS ECS/Fargate	Medium	$50-300+	Auto-scaling with AWS complexity	Works until vendor lock-in issues
Google Cloud Run	Low	Pay-per-request (expensive at scale)	Serverless automatic	Cost explosion under high load
Railway/Render	Very Low	$5-25 (until scaling needs)	Limited scaling capacity	Hits limits quickly, good for MVPs only

Critical Production Warnings

Security Issues

• Environment variable secrets: Visible in ps aux - not actual secrets management
• CORS production failures: Never use .allow_any_origin() in production - specify exact domains
• File upload security: Don't trust MIME types, implement size limits, use external storage (S3/R2)

Database Migration Failures

• SQLx compile-time check breakage: Run sqlx migrate run then cargo sqlx prepare for offline mode
• Zero-downtime requirement: All migrations must be backward-compatible - add nullable columns, never remove in same deploy

Monitoring Critical Failures

• High-cardinality Prometheus labels: Will crash Prometheus server with memory exhaustion
• Debug logging disk fill: Filled 50GB in hours with SQL query logs - use INFO level only
• Log rotation necessity: Implement centralized logging and retention policies

Graceful Shutdown Requirements

• Signal handling: Must implement SIGTERM handling with tokio::signal
• Shutdown timeout: 30-60 seconds (too short drops connections, too long delays deployments)
• Rolling deployment reality: "Zero-downtime" fails ~15% of the time - plan for this

Operational Intelligence

What Official Documentation Doesn't Cover

• Docker networking doesn't resolve localhost in containers
• Alpine containers have musl libc compatibility issues causing random crashes
• SQLx offline mode required for migrations in production builds
• Prometheus memory usage scales exponentially with metric label cardinality
• Health checks run constantly from multiple load balancer nodes

Time Investment Reality

• Initial deployment setup: 1-2 days for experienced developers
• Debugging production networking issues: 3-6 hours typical
• Setting up monitoring stack: 4-8 hours
• Migration to production-ready configuration: 2-3 iterations of complete rebuilds

Breaking Points and Thresholds

• Memory: Apps killed silently when exceeding container limits
• Database connections: Pool exhaustion causes request timeouts with minimal error information
• Prometheus: Server crashes when metric cardinality exceeds memory capacity
• Log volume: Debug level logging can fill 50GB+ in hours under load

Community and Support Quality

• Rust ecosystem: Excellent performance, steep deployment learning curve
• Docker with Rust: Multi-stage builds mandatory, documentation gaps for production
• Kubernetes: Powerful but operationally expensive for small teams
• Cloud platforms: Good reliability, vendor lock-in concerns, costs scale quickly

Error Patterns and Root Causes

Database Connection Issues

• Symptom: Intermittent timeouts
• Root cause: Connection pool too small or queries too slow
• Solution: Monitor pool metrics, tune min/max connections based on actual usage

Container Memory Kills

• Symptom: Mysterious app deaths with no clear logs
• Root cause: Container memory limits exceeded
• Solution: Set realistic memory limits, monitor usage patterns under load

Health Check Database Overload

• Symptom: Database performance degradation
• Root cause: Multiple load balancers hitting health endpoint constantly
• Solution: Separate lightweight readiness checks from thorough health checks

Prometheus Memory Exhaustion

• Symptom: Monitoring server crashes or becomes unresponsive
• Root cause: High-cardinality metric labels (user IDs, request IDs)
• Solution: Use sampling, implement cardinality limits, avoid unique identifiers as labels