Nix Production Deployment: AI-Optimized Technical Reference

Executive Summary

Nix provides immutable infrastructure with atomic deployments and zero-downtime rollbacks. Three deployment approaches exist, with clear progression paths and specific failure modes. Binary caches are mandatory for production - deployments without them take 2-4 hours instead of 2-5 minutes.

Deployment Approaches: Comparative Analysis

Direct nixos-rebuild (Development Only)

Configuration: SSH to server, edit /etc/nixos/configuration.nix, run nixos-rebuild switch
Resource Requirements:

• Time: 30 seconds deployment, 2-5 minutes panic recovery
• Expertise: 5 minutes learning curve
• Team limit: 1-2 people maximum

Critical Warnings:

• Server configuration drift inevitable with multiple servers
• No audit trail or rollback capability
• Manual panic recovery only
• Breaks at 2+ servers due to consistency issues

Use Case: Single server, low traffic, solo developer, infrequent changes

Remote nixos-rebuild (Small Production)

Configuration:

nixos-rebuild switch \
  --build-host build-server.example.com \
  --target-host prod-server.example.com \
  --use-remote-sudo

Resource Requirements:

• Time: 5-15 minutes (serial deployment)
• Expertise: Weekend to understand properly
• Team limit: 2-3 people before conflicts
• Server limit: 2-10 servers before unmanageable

Critical Warnings:

• Serial deployment creates extended vulnerability windows
• Build server mandatory (1-CPU production builds kill service for 3 hours)
• Manual rollback requires SSH to each server
• Binary cache misconfiguration causes source builds during peak traffic

Breaking Points:

• More than one person deploying causes chaos
• Firefox compilation on production server: 3+ hour outage
• Cache failures during traffic spikes: complete service degradation

Deploy-rs + Flakes (Production Standard)

Configuration:

deploy.nodes.web-server = {
  hostname = "web01.prod.example.com";
  profiles.system = {
    user = "root";
    path = deploy-rs.lib.x86_64-linux.activate.nixos 
      self.nixosConfigurations.web-server;
  };
};

Resource Requirements:

• Time: 2-5 minutes for 100+ servers (parallel deployment)
• Expertise: 2-3 days initial learning
• Team scalability: Unlimited
• Infrastructure: Scales to 100+ servers

Production Features:

• Parallel deployment across all servers
• Atomic rollback if any server fails
• Magic rollback: automatic revert if SSH breaks (30 seconds)
• Multi-profile support for non-root deployments
• Interactive preview mode

Deployment Command: deploy .

Binary Cache Strategy (Mandatory for Production)

Cache Hit Rate Requirements

• Minimum acceptable: 80% hit rate
• Standard production: 90%+ hit rate
• Monitoring imperative: Alert on <80% hit rate

Cache Options with Cost Analysis

Solution	Cost	Use Case	Reliability
cache.nixos.org	Free	Development, covers 95% nixpkgs	External dependency risk
Cachix	$45/month	Production, custom packages	Commercial SLA
FlakeHub Cache	Enterprise pricing	Large orgs, private flakes	Enterprise support
Self-hosted Attic	Infrastructure cost	Full control, compliance	Self-managed

Cache Failure Impact

• Without cache: 2-4 hour deployments (source compilation)
• With cache: 2-5 minute deployments
• Firefox build example: 8GB+ RAM usage on 2GB server = complete system failure

Critical Production Gotchas

/nix/store Disk Space Disaster

Failure Mode: Root filesystem fills up, server stops accepting connections
Root Cause: No automatic garbage collection of old generations
Detection: df -h /nix/store shows >90% usage
Emergency Fix: nix-collect-garbage --delete-older-than 3d
Prevention:

nix.gc = {
  automatic = true;
  dates = "weekly";
  options = "--delete-older-than 30d";
};

Real Impact: Client's Black Friday checkout failure (45GB old generations)

Binary Cache Authentication Failure

Failure Mode: Silent fallback to source compilation, 3+ hour deployments
Detection:

nix store ping --store https://cache.nixos.org
nix build --print-build-logs --verbose

Monitoring: Alert on cache hit rate <80%

SSH Permission Denied with deploy-rs

Failure Signature: ssh -o BatchMode=yes fails but ssh works
Root Cause: SSH agent not available to deploy-rs
Fix:

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa
export SSH_AUTH_SOCK

Flake Input Pinning Disasters

Failure Mode: Working builds fail overnight due to unpinned inputs
Root Cause: nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable" pulls latest
Prevention: Pin everything in flake.lock, update explicitly only
Real Example: systemd regression from unpinned input caused 4-hour debugging session

Build User Exhaustion

Failure Mode: "waiting for build users" during high activity
Root Cause: Default 32 build users insufficient under load
Solution: Increase build user count and set max-jobs = 8
Monitoring: ps aux | grep nix-daemon | wc -l

Memory Exhaustion During Builds

Failure Mode: System lockup, no SSH/HTTP response
Root Cause: Large package builds (Firefox, browsers) on memory-constrained servers
Prevention: Use separate build server or add 8GB+ swap
Real Impact: 2GB server building Firefox = 4-hour complete outage

CI/CD Integration Patterns

Production-Grade GitHub Actions

- uses: DeterminateSystems/nix-installer-action@v4
- uses: DeterminateSystems/magic-nix-cache-action@v2
- name: Build system configurations  
  run: nix build '.#nixosConfigurations.web-server.config.system.build.toplevel'
- name: Deploy to production
  run: deploy . --skip-checks

Performance Benchmarks:

• Build to deployment: 5-8 minutes total
• Docker equivalent: 20-30 minutes
• Magic Nix Cache significantly accelerates CI builds

Secrets Management Strategies

NEVER: Put secrets in Nix store (world-readable)
Development: sops-nix with repository encryption
Production: External stores (Vault, AWS Secrets Manager)
Simple approach:

systemd.services.myapp = {
  serviceConfig.EnvironmentFile = "/etc/secrets/myapp.env";
};

Monitoring and Alerting Requirements

Disk Space Monitoring

alert = "NixStoreDiskFull";
expr = "disk_free_bytes{mountpoint=\"/nix/store\"} < 10000000000";  # <10GB

Cache Performance

• Monitor cache hit rates
• Alert on <80% hit rate
• Track deployment duration trends

Build Resource Usage

• Monitor build user utilization
• Track memory consumption during builds
• Alert on build timeouts >10 minutes

Enterprise Production Evidence

Successful Deployments

• FlightAware: Flight tracking infrastructure, self-hosted caches
• Shopify: Developer environments, build tooling
• IOHK: Cardano blockchain infrastructure
• Tweag: Client consulting infrastructure

Performance Comparisons

Metric	Kubernetes	Nix + deploy-rs
Deployment Time	15-25 minutes	3-5 minutes
Partial Failures	Common	Impossible (atomic)
Rollback Time	5-10 minutes	30 seconds
Configuration Drift	Frequent	Impossible

Resource Requirements by Scale

Small Production (2-10 servers)

• Deployment time: 5-15 minutes (serial)
• Cache requirement: Cachix or cache.nixos.org
• Team size: 2-3 developers maximum
• Approach: Remote nixos-rebuild

Large Production (10+ servers)

• Deployment time: 2-5 minutes (parallel)
• Cache requirement: Self-hosted or enterprise cache
• Team size: Unlimited
• Approach: Deploy-rs + flakes mandatory

Implementation Progression Strategy

1. Week 1: Development environments with nix-shell
2. Week 2-3: Add flake.nix to one repository
3. Month 1: Deploy to staging with deploy-rs
4. Month 2: Migrate production one service at a time

Critical: Never attempt full migration in single weekend - causes system-wide failures

Decision Criteria Matrix

Use Direct nixos-rebuild When:

• Single server deployment
• Solo developer environment
• Infrequent configuration changes
• Learning/development phase

Use Remote nixos-rebuild When:

• 2-10 servers requiring coordination
• Manual deployment acceptable
• Small team (2-3 people)
• Intermediate complexity tolerance

Use Deploy-rs + Flakes When:

• 10+ servers or mission-critical systems
• Team deployment requirements
• Zero-downtime deployment mandatory
• Enterprise compliance needed

Failure Recovery Procedures

Emergency Disk Space Recovery

nix-collect-garbage --delete-older-than 1d  # Emergency only
systemctl restart nix-daemon

Cache Failure Recovery

# Verify cache connectivity
nix store ping --store https://your-cache.com
# Force rebuild with verbose logging
nix build --rebuild --print-build-logs

Deployment Rollback

# Automatic with deploy-rs
deploy . --magic-rollback true
# Manual approach
nixos-rebuild --rollback

This technical reference provides the operational intelligence needed for successful Nix production deployments, including all critical failure modes, resource requirements, and proven implementation strategies.