Deploy Axum Apps to Production Without Losing Your Mind

Prerequisites and Production Environment Setup

I've deployed a bunch of Axum apps since 0.8 dropped. Here's what I learned getting burned by production over the last few months.

Why Production Will Break Your Beautiful Local Setup

Your Axum app works perfectly on localhost. It starts in 100ms, handles requests beautifully, and the logs are pristine. Then you deploy it and everything goes to shit.

Production is where your beautiful local setup goes to die. I've lost entire weekends debugging why my app crashes with "Connection refused" when it runs fine in development. The problem? Docker's networking doesn't resolve localhost - you need to use the service name or 0.0.0.0. Axum itself is solid but the surrounding ecosystem will kick your ass.

Real production deployments involve shit that doesn't happen locally: network timeouts, database connection pool exhaustion, memory limits that kill your process, and load balancers that decide your health checks are lying. Yeah, Discord and Dropbox use Rust for performance. Doesn't mean deployment stops being a pain in the ass.

The Shit You Actually Need (Not the Tutorial Stuff)

System Requirements

You need Linux. I've tried Windows containers - don't. Any recent Ubuntu or Debian works fine. Docker that isn't ancient. Minimum 512MB RAM, but budget 2GB because your app will eat way more memory than you expect. Learned this after my app kept crashing every few hours with no clear logs. Spent a weekend thinking it was a Rust memory leak or some database connection issue. Finally found buried in the system logs that Docker was killing it for memory usage. Classic.

Rust Toolchain

Install current stable Rust. Don't get fancy with nightly - production deployment is complicated enough. Use multi-stage Docker builds because compiling Rust apps takes forever and the final binary is tiny compared to the build environment.

Development Environment

Your local environment should mirror production or you'll spend hours debugging differences. Use Docker Compose locally - I don't care if you prefer running PostgreSQL natively, containers save your sanity. The dotenv crate works for local env management, but don't use .env files in production.

Database Integration

Most production Axum apps need PostgreSQL with SQLx for compile-time query verification. This is brilliant when it works and absolutely infuriating when it doesn't. Migrations break SQLx compile-time checks in weird ways, and you'll spend an hour figuring out offline mode exists.

Redis works great for caching until your connection pool settings are fucked and requests start hanging with no clear error messages.

External Dependencies

Third-party APIs will fail at the worst possible time. Set aggressive timeouts - I default to 10 seconds max. The reqwest crate with connection pooling prevents the "too many open files" nightmare. Consider circuit breaker patterns for failing services and retry strategies for transient failures.

Configuration That Won't Screw You Later

Never hardcode anything. Seriously. Use environment variables for everything deployment-specific. The config crate is decent for structured configuration, but envy is simpler for deserializing environment variables into structs.

Secrets Management

Environment variables visible to ps aux are not secrets management. Use Kubernetes secrets if you're on K8s, Docker secrets for Swarm, or cloud provider solutions like AWS Secrets Manager. I've seen production API keys in git history - don't be that person.

Health Checks

You need /health and /ready endpoints. Load balancers and orchestrators depend on these. Make them actually check your dependencies - a health check that always returns 200 is useless. But don't make health checks expensive or your load balancer will kill healthy instances during traffic spikes. Consider implementing liveness vs readiness checks properly if using Kubernetes.

The rest of this guide covers the details that will save you from 3am debugging sessions. The Twelve-Factor App methodology has more production deployment principles that actually matter, if you're into that sort of thing.

Docker: Where Everything That Can Go Wrong, Will

Docker is great in theory. In practice, you'll spend 3 hours debugging why your perfectly working local Rust app crashes with SIGKILL in a container, and the error messages will be completely useless. Multi-stage builds are mandatory - without them, your Docker image will be 1.5GB of Rust toolchain that nobody needs in production.

This Dockerfile Actually Works (After 6 Attempts)

Creating Docker images for Rust is painful. Your first attempt will fail to find system libraries. Your second will work locally but crash in production. Your third will build successfully but take 45 minutes every time. Here's the Dockerfile that finally worked:

## This Dockerfile took me like 6 or 7 attempts. I stopped counting after the third Alpine disaster.
FROM rust:slim-bookworm AS builder

## These deps are required or your build will fail with cryptic errors
RUN apt-get update && apt-get install -y pkg-config libssl-dev && rm -rf /var/lib/apt/lists/*

WORKDIR /app

## Layer caching trick - copy deps first, then rebuild only app code when changed
COPY Cargo.toml Cargo.lock ./
RUN mkdir src && echo "fn main() {}" > src/main.rs
RUN cargo build --release && rm -rf src

COPY src ./src
## Force rebuild - without this, cargo thinks nothing changed
RUN touch src/main.rs && cargo build --release

## Runtime image - debian because Alpine breaks with musl linking issues
FROM debian:bookworm-slim
RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/*

## Don't run as root - basic security 
RUN useradd --create-home app
COPY --from=builder /app/target/release/your-app /usr/local/bin/
USER app

EXPOSE 8080
CMD ["your-app"]

Why This Works: The layer caching trick saves your sanity. Without copying Cargo.toml first, every code change rebuilds all your dependencies - learned this after waiting forever for SQLx and OpenSSL to recompile because I changed a single line.

Security Notes: Don't run as root in containers. Debian base is larger than Alpine but Alpine has weird musl libc issues that will bite you. I wasted an entire Saturday debugging random segfaults that only happened in Alpine containers. Stack traces were useless. Turned out to be some musl vs glibc nonsense that I never fully understood. Use hadolint to lint your Dockerfiles.

Cargo.toml That Won't Embarrass You in Production

Your development build settings are probably wrong for production. Here's what you need for builds that don't suck:

[profile.release]
lto = true          # Makes binary smaller and faster, takes longer to compile
codegen-units = 1   # Better optimization, much slower compile time
panic = "abort"     # Don't unwind on panic, smaller binary  
strip = true        # Remove debug symbols, saves space

[dependencies]
axum = { version = "0.8", features = ["macros"] }
tokio = { version = "1.0", features = ["full"] }
tower = "0.4" 
tower-http = { features = ["cors", "compression", "trace"] }
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
serde = { version = "1.0", features = ["derive"] }

## SQLx with PostgreSQL - this will save your ass with compile-time query checking
[dependencies.sqlx]
version = "0.7"
features = ["runtime-tokio-rustls", "postgres", "chrono", "uuid"]

## Don't include metrics in every build - feature flag it
[features]
default = []
metrics = ["dep:metrics", "dep:metrics-prometheus"]

[dependencies.metrics]
version = "0.23"
optional = true

[dependencies.metrics-prometheus]
version = "0.7" 
optional = true

Build Time Reality: Link-time optimization makes your binary 20% smaller and faster, but compile times go from 2 minutes to 8 minutes. Worth it for production, annoying for development. Consider using cargo-chef for faster Docker builds and sccache for build caching.

Health Checks That Don't Lie

Your health check endpoint is critical. Load balancers, Kubernetes, and monitoring systems depend on it. A health check that always returns 200 OK is fucking useless.

use axum::{extract::State, http::StatusCode, routing::get, Json, Router};
use serde_json::json;

// This health check actually does something useful
async fn health_check(State(app_state): State<AppState>) -> Result<Json<serde_json::Value>, StatusCode> {
    // Check database - if this fails, your app is fucked anyway
    match sqlx::query("SELECT 1").execute(&app_state.db_pool).await {
        Ok(_) => Ok(Json(json!({
            "status": "healthy",
            "database": "connected",
            "timestamp": chrono::Utc::now()
        }))),
        Err(e) => {
            tracing::error!("Health check failed: {}", e);
            Err(StatusCode::SERVICE_UNAVAILABLE)
        }
    }
}

// Kubernetes readiness check - different from health check
async fn readiness_check() -> StatusCode {
    // Don't check database here - use this for "ready to receive traffic"
    StatusCode::OK
}

pub fn health_routes() -> Router<AppState> {
    Router::new()
        .route("/health", get(health_check))
        .route("/ready", get(readiness_check))
}

Health Check Reality: I learned the hard way that health checks run constantly. If your health check hits the database every 2 seconds from 10 load balancer nodes, you're doing 50 database queries per second just for health checks. Make /ready lightweight and /health thorough.

Resource Limits: Set memory limits or Kubernetes will kill your container when it uses too much. Rust apps are memory-efficient but not magic - my first production deploy kept getting mysteriously killed during traffic spikes. No errors, no logs, just dead. Took me way too long to realize I'd set a 256MB limit and the app was using 400MB under load. Kubernetes doesn't tell you this stuff clearly.

Graceful Shutdown: Your app needs to handle SIGTERM gracefully. Without this, rolling deployments will drop connections and users will see errors. Use axum::Server::with_graceful_shutdown() and actually implement signal handling with tokio::signal. The shutdown crate provides additional utilities for coordinating shutdown sequences.

Next up: monitoring, because you'll need to know when everything breaks at 3am. Consider implementing proper observability with distributed tracing from the start.

You need monitoring or you'll be blind when your app breaks in production. And it will break. The question is whether you'll know about it from monitoring alerts or angry customers.

Tracing: Logging That Doesn't Suck

Tracing is Rust's answer to structured logging. It's better than println! debugging, and you can actually search and analyze the output. Here's how to set it up without losing your mind:

use axum::{extract::State, routing::post, Json, Router};
use tracing::{info, instrument, error};
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

// This tracing setup works in production
pub fn init_tracing() {
    tracing_subscriber::registry()
        .with(
            tracing_subscriber::EnvFilter::try_from_default_env()
                .unwrap_or_else(|_| "info".into()),
        )
        .with(tracing_subscriber::fmt::layer().json())
        .init();
}

#[instrument(skip(state), fields(user_id))]
async fn create_user(
    State(state): State<AppState>,
    Json(request): Json<CreateUserRequest>,
) -> Result<Json<User>, AppError> {
    info!("Creating new user: {}", request.email);
    
    match sqlx::query_as!(User, "INSERT INTO users (name, email) VALUES ($1, $2) RETURNING *", 
                        request.name, request.email)
        .fetch_one(&state.db_pool)
        .await 
    {
        Ok(user) => {
            info!(user_id = user.id, "User created successfully");
            Ok(Json(user))
        }
        Err(e) => {
            error!(error = %e, "Database insert failed: {}", e);
            Err(AppError::DatabaseError)
        }
    }
}

JSON Logging Reality: Use JSON format in production so your log aggregation system can parse it. The #[instrument] macro is magic - it automatically adds timing and context to every function call. But don't instrument every function or your logs will be 90% noise.

Log Levels: Set RUST_LOG=info in production. Debug logging killed our staging server - filled like 50GB in a few hours with SQL query logs. Fun times explaining that to ops. Use log rotation and centralized logging to manage log volume.

Prometheus Metrics: Numbers That Matter

Logs tell you what happened, metrics tell you how fucked you are. Prometheus and Grafana are the standard for production monitoring - learn them or debug blindfolded.

HTTP Metrics: tower-http gives you free HTTP metrics. Use them or you'll be guessing why your API is slow. The metrics crate provides the core primitives, while metrics-exporter-prometheus handles the Prometheus integration. Consider axum-prometheus for easier setup:

use tower_http::{trace::TraceLayer, metrics::InFlightRequestsLayer};
use metrics_exporter_prometheus::PrometheusBuilder;

// This works but takes some setup
pub fn init_metrics() -> Result<(), Box<dyn std::error::Error>> {
    PrometheusBuilder::new()
        .with_http_listener([0, 0, 0, 0], 9090)
        .install()?;
    Ok(())
}

let app = Router::new()
    .route("/users", post(create_user))
    .layer(TraceLayer::new_for_http())
    .layer(InFlightRequestsLayer::new())
    .with_state(app_state);

Metrics Reality: Prometheus metrics are great until you have 10,000 metrics labels and your Prometheus server runs out of memory. Be careful with high-cardinality labels like user IDs - this will bite you hard when Prometheus starts eating all your RAM and you can't figure out why. Use sampling and cardinality limits or prepare for pain. The OpenTelemetry Rust SDK provides more advanced instrumentation options.

Error Tracking: Logs and metrics aren't enough for production. You need proper error tracking with Sentry or similar. The sentry crate integrates well with Axum. Use anyhow for better error handling and thiserror for custom error types. Don't forget to implement proper error responses - users shouldn't see "Internal Server Error" with no context.