Azure Container Instances Production Troubleshooting - Fix the Shit That Always Breaks

When your ACI containers are down and you're getting paged at 3am, you don't have time for Microsoft's 47-step troubleshooting flowchart. Here's the methodology that actually works when everything's on fire.

Step 1: Get the Real Error Message (Not the Bullshit Summary)

The Azure portal shows you friendly error messages that are completely useless. Get the actual error details:

az container show --resource-group mygroup --name mycontainer --query \"containers[0].instanceView\"

This gives you the real container events, not the sanitized portal version. Look for the previousState and currentState sections - that's where the actual errors hide.

What you're looking for: Exit codes, specific error messages, and the restartCount. If restartCount keeps climbing, you have a restart loop. If it's stuck at 0 with state: Waiting, your container never started properly.

Step 2: The Resource Check (Because Azure Lies About Availability)

ACI will tell you resources are available in a region, then fail to provision them. Check what you're actually asking for:

## See what you requested
az container show --resource-group mygroup --name mycontainer --query \"containers[0].resources.requests\"

## See what you actually got (if anything)
az container show --resource-group mygroup --name mycontainer --query \"containers[0].resources.limits\"

If you requested 4 vCPUs and got nothing, try 2 vCPUs. If you requested 8GB RAM and got nothing, try 4GB. ACI availability is unpredictable, especially in smaller regions.

Regional reality check: Some Azure regions just don't have capacity for larger container instances. If you're stuck, try these regions that usually have availability:

• East US 2
• West Europe
• Southeast Asia

Step 3: The Image Pull Investigation

Image pull failures are the most common production issue because there are 15 different ways they can break:

Registry connectivity test:

az acr login --name myregistry
docker pull myregistry.azurecr.io/myapp:latest

If this fails, your image doesn't exist or your credentials are wrong. If it works, ACI is having registry authentication issues.

Registry authentication debugging:

## Check ACR credentials
az acr credential show --name myregistry

## Check managed identity assignment
az container show --resource-group mygroup --name mycontainer --query \"identity\"

The nuclear option: If nothing else works, copy your image to Docker Hub temporarily. It's not secure for production, but it'll get your service back up while you fix ACR authentication:

docker tag myregistry.azurecr.io/myapp:latest mydockerhubuser/myapp:emergency
docker push mydockerhubuser/myapp:emergency

Step 4: Container Startup Debugging

If your container pulls successfully but crashes immediately, you need to debug the startup process:

Method 1: Override the entrypoint

az container create --resource-group mygroup --name debug-myapp \
  --image myregistry.azurecr.io/myapp:latest \
  --command-line \"tail -f /dev/null\"

This keeps the container running so you can exec in and debug manually.

Method 2: Check environment variables

az container exec --resource-group mygroup --container-name debug-myapp --exec-command \"env\"

Your app might be looking for environment variables that aren't set. Production containers often have different ENV requirements than development.

Method 3: Run your app manually

az container exec --resource-group mygroup --container-name debug-myapp --exec-command \"/bin/bash\"
## Then inside the container
/path/to/your/app

This will show you the actual error your app throws, which is usually more helpful than ACI's generic error messages.

Step 5: The Memory and CPU Reality Check

ACI enforces resource limits strictly. If your app works fine in development but crashes in ACI, you're probably hitting resource limits:

Memory debugging:

az container exec --resource-group mygroup --container-name mycontainer --exec-command \"free -h\"

## Check what processes are using memory
az container exec --resource-group mygroup --container-name mycontainer --exec-command \"ps aux --sort=-%mem\"

The 25% rule: Always request 25% more CPU and memory than your app needs. ACI doesn't handle resource contention gracefully - it just kills containers that exceed limits.

CPU performance gotcha: Fractional vCPUs (0.5, 1.5) perform poorly when Azure is busy. If your app is CPU-intensive, use whole vCPU numbers (1, 2, 4).

This methodology has saved me countless hours of frustration. Skip the Microsoft docs flowcharts and go straight to what actually identifies the problem.

Most ACI tutorials show you hello-world demos that work perfectly in development but fall apart the moment real users hit them. Here are the patterns that survive production deployment.

The "Always-On" Container Pattern

ACI containers can randomly restart for platform maintenance. Your application needs to handle this gracefully or you'll have mysterious 30-second outages.

What doesn't work: Storing application state in memory or local files. When ACI restarts your container, everything in memory disappears.

What works: Store state externally and design for quick restart:

## Bad: state stored in container
webapp_container:
  image: myapp:latest
  # App stores user sessions in memory
  # File uploads saved to /tmp
  # Database connections cached in global variables

## Good: stateless design
webapp_container:
  image: myapp:latest
  environment:
    REDIS_URL: myredis.redis.cache.windows.net
    BLOB_STORAGE: mystorageaccount.blob.core.windows.net
    DB_CONNECTION_POOL_SIZE: 10

Startup time reality: ACI cold starts can take 15-60 seconds (or forever on a bad day). Speed up your container startup:

• Use Alpine Linux base images (5MB vs 200MB for Ubuntu)
• Pre-compile languages like .NET with ReadyToRun
• Cache dependencies in the Docker image, not downloaded at startup
• Use multi-stage builds to minimize final image size

The "Scale-to-Zero" Anti-Pattern

ACI markets itself as "scale to zero" but this only works for batch jobs. Web applications that scale to zero have terrible user experience.

The problem: First request after scale-to-zero takes 30-90 seconds to respond while ACI provisions a new container and your app starts up.

Better approach: Use 1 minimum instance with Azure Container Apps instead:

az containerapp create --resource-group mygroup --name myapp \
  --image myapp:latest \
  --min-replicas 1 --max-replicas 10

Container Apps gives you the serverless benefits without the cold start penalty for the first user.

When ACI scale-to-zero makes sense:

• Scheduled batch jobs (run at 2am, complete in 10 minutes, shut down)
• CI/CD build agents (spin up per build, shut down after completion)
• Development environments (nobody cares about 30-second startup times)

The "Monitoring That Actually Helps" Pattern

Azure's default ACI monitoring shows you pretty graphs that don't help during outages. Set up monitoring that tells you what's broken and how to fix it.

Essential metrics to track:

## Container restart count (alerts when > 3 in 10 minutes)
az monitor metrics list --resource mycontainer --metric restartCount

## Memory utilization (alerts when > 80% to prevent OOM kills)
az monitor metrics list --resource mycontainer --metric memoryUsage

## CPU utilization over time (identifies performance degradation)
az monitor metrics list --resource mycontainer --metric cpuUsage

Log aggregation that doesn't suck: ACI's default logging only keeps logs for 7 days. Send logs somewhere useful:

az container create --resource-group mygroup --name mycontainer \
  --image myapp:latest \
  --log-analytics-workspace /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.OperationalInsights/workspaces/myworkspace

Error alerting: Set up alerts for actual error conditions, not just "container restarted":

## Alert when container fails to start 3 times in 15 minutes
az monitor scheduled-query create --resource-group mygroup \
  --name \"ACI-StartupFailures\" \
  --scopes mycontainer \
  --query \"ContainerInstanceLog_CL | where TimeGenerated > ago(15m) | where LogEntry_s contains 'Failed to pull image' | summarize count() by bin(TimeGenerated, 5m) | where count_ >= 3\"

The "Cost Control That Prevents Bill Shock" Pattern

ACI's per-second billing sounds cheap until you realize a container stuck in a restart loop can cost hundreds of dollars before anyone notices.

Resource limits that prevent runaway costs:

az container create --resource-group mygroup --name mycontainer \
  --image myapp:latest \
  --cpu 1 --memory 2 \
  --restart-policy OnFailure  # Don't restart forever
  --max-restart-count 3       # Give up after 3 failures

Regional cost optimization: ACI pricing varies significantly by region. For batch workloads, use cheaper regions:

• East US: $0.045/vCPU-hour
• South Central US: $0.043/vCPU-hour
• West Europe: $0.054/vCPU-hour
• Japan East: $0.061/vCPU-hour

Spot container risk management: Spot instances save money but can disappear. Use them for fault-tolerant workloads:

## Spot container with automatic failover to regular pricing
az container create --resource-group mygroup --name myapp-spot \
  --image myapp:latest --priority Spot

## If spot fails, create regular container as backup
az container create --resource-group mygroup --name myapp-regular \
  --image myapp:latest --priority Regular

These patterns come from learning ACI's limitations the hard way. Use them to avoid the common production failures that turn simple container deployments into 3am emergency calls.