Container keeps restarting with exit code 137 - what the hell is happening?

Your container died with exit code 137? Yeah, that's the kernel killing it for eating too much memory. Been there, it sucks. ACI enforces [memory limits](https://www.groundcover.com/kubernetes-troubleshooting/exit-code-137) like a bouncer - no warnings, just instant death when you cross the line. **Quick fix**: ```bash az container show --resource-group mygroup --name mycontainer --query "containers[0].resources" ``` If your container is requesting 1GB but actually needs 1.5GB, bump it to 2GB. Don't try to be clever with exact limits - Azure bills per second so the extra memory costs pennies compared to downtime. **Why this happens**: Your Node.js app that "only uses 500MB in dev" suddenly needs 2GB in production because you forgot about the 15 npm packages that cache everything in memory. Or that Python script that seemed fine locally but loads the entire dataset into a pandas DataFrame on Azure.

"Failed to pull image" but the image exists and worked yesterday

This is the most frustrating error because it's usually not your fault. ACI's image pulling is inconsistent as hell, especially for private registries. **First thing to try**: Delete and recreate the container group. Seriously. ACI gets stuck in weird states. ```bash az container delete --resource-group mygroup --name mycontainer --yes az container create --resource-group mygroup --name mycontainer --image myimage ``` **If that doesn't work**: Check if your ACR credentials expired. Service principals have expiration dates that nobody remembers to renew: ```bash az acr credential show --name myregistry ``` **Last resort**: Use managed identity instead of service principals. At least then Azure handles the credential rotation: ```bash az container create --resource-group mygroup --name mycontainer \ --image myregistry.azurecr.io/myapp:latest \ --assign-identity --acr-login-server myregistry.azurecr.io ```

Container starts then immediately dies - no useful logs

This usually means your app is crashing before it can write anything meaningful. The container starts, your process exits, ACI restarts it, repeat forever. **Debug with a sleep command**: Override the container command to keep it running so you can exec into it: ```bash az container create --resource-group mygroup --name debug-container \ --image myapp:latest --command-line "sleep 3600" ``` Then exec in and run your app manually: ```bash az container exec --resource-group mygroup --container-name debug-container --exec-command "/bin/bash" ``` **Common causes**: - Missing environment variables your app needs - Database connection strings that don't work from Azure - File permissions fucked up in your Docker image - Your app expects files that don't exist in the container

Container group stuck in "Pending" state forever

This means Azure can't find resources to run your container. Usually happens in smaller regions or when you're asking for too much CPU/memory. **Quick fix**: Try a different region: ```bash az container create --resource-group mygroup --name mycontainer \ --image myapp:latest --location eastus2 ``` **Resource requirements check**: ACI has quotas per region. If you're asking for 4 vCPUs in West Europe during peak hours, you might wait forever. Drop to 2 vCPUs or try a different region. **Emergency workaround**: Use spot containers if your workload can handle interruptions: ```bash az container create --resource-group mygroup --name mycontainer \ --image myapp:latest --priority Spot ```

Networking issues - can't connect to container ports

ACI networking is simple until it isn't. Most connection issues come down to port mismatches or security group bullshit. **Port mapping reality check**: Unlike Docker, ACI doesn't do port mapping. If your app listens on port 3000 inside the container, you expose port 3000 in ACI. Period. ```bash az container create --resource-group mygroup --name mycontainer \ --image myapp:latest --ports 3000 --ip-address Public ``` **VNet deployment gotcha**: If you deploy into a VNet, you need a NAT gateway for outbound connections. Microsoft's docs mention this in small print after you've already hit the issue: ```bash # Your container can't reach the internet without this az network nat gateway create --resource-group mygroup --name mynatgateway ```

Multi-container groups: one container kills the others

Container groups in ACI are like a house of cards. When one container in the group crashes, it can take down the whole group depending on your restart policy. **The problem**: You have a web app + Redis sidecar. Redis runs out of memory and crashes. ACI restarts the entire container group, killing your web app sessions. **Quick fix**: Separate your containers into different container groups: ```bash # Create Redis separately az container create --resource-group mygroup --name redis-instance \ --image redis:alpine --memory 1 --cpu 0.5 # Create web app that connects to Redis via its IP az container create --resource-group mygroup --name webapp-instance \ --image myapp:latest --memory 2 --cpu 1 \ --environment-variables REDIS_HOST= ``` **When you actually need container groups**: Only when containers absolutely must share localhost networking. Otherwise, separate container groups are more reliable.

Storage volumes randomly unmount

ACI's [Azure Files integration](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-volume-azure-files) looks great on paper but fails in creative ways in production. **Symptoms**: Your app writes to `/data/logs/app.log` successfully, then later crashes because the file doesn't exist. The Azure Files share appears to be unmounted. **Root cause**: Network hiccups between ACI and Azure Files. When the network connection drops, the mount fails silently. **Workaround**: Mount Azure Files with error handling in your application: ```bash # Check if mount exists before writing if [ -d \"/data\" ]; then echo \"Writing to /data/logs/app.log\" else echo \"Mount failed, writing to /tmp/app.log instead\" fi ``` **Better solution**: Use blob storage with the Azure CLI instead of file mounts: ```bash az storage blob upload --account-name mystorage --container-name logs \ --name \"app-$(date +%Y%m%d).log\" --file /tmp/app.log ```

Confidential containers cost a fortune but don't work with your app

Microsoft's [confidential containers](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-confidential-overview) sound cool until you realize they break half your applications and cost 3x normal pricing. **What breaks**: - Apps that need privileged operations - Containers that access hardware directly - Some Node.js modules that use native binaries - Any app that expects specific CPU features **Cost reality**: Confidential containers can cost $0.15/vCPU-hour vs $0.045/vCPU-hour for regular containers. That's $108/month vs $32/month for a single 1-vCPU container running 24/7. **When to actually use them**: Processing truly sensitive data where the extra cost is worth it. For most applications, standard ACI with network security groups is sufficient.

Spot containers get evicted during critical operations

[Spot containers](https://learn.microsoft.com/en-us/azure/container-instances/container-instances-spot-containers-overview) save 70% on costs but can disappear with 30 seconds notice when Azure needs the capacity back. **The problem**: You're running a 4-hour batch job on spot containers. After 3 hours and 45 minutes, Azure evicts your container and you lose all progress. **Solution pattern**: Checkpoint your progress regularly: ```bash # Save state every 15 minutes while true; do process_batch_chunk() save_state_to_blob() sleep 900 done ``` **Don't use spot for**: - Web applications (users will get 500 errors when evicted) - Database containers - Long-running tasks without checkpointing **Good for**: - CI/CD pipelines (can restart from failed step) - Log processing (can reprocess failed batch) - Development/testing environments

Regional capacity issues during Azure outages

When Azure has regional issues, ACI often fails silently. Your container group just sits in \"Pending\" state forever. **Detection**: If your container has been pending for more than 10 minutes, it's probably a capacity issue: ```bash az container show --resource-group mygroup --name mycontainer --query \"provisioningState\" ``` **Multi-region deployment**: Deploy identical container groups in multiple regions: ```bash # Primary region az container create --resource-group mygroup-east --name myapp-east \ --image myapp:latest --location eastus2 # Backup region az container create --resource-group mygroup-west --name myapp-west \ --image myapp:latest --location westus2 ``` Use Azure Traffic Manager to route traffic between healthy instances. It's more expensive but prevents total outages during regional issues.

Currently viewing the AI version

Switch to human version

Azure Container Instances: Production Troubleshooting Reference

Critical Failure Scenarios and Fixes

Container Exit Code 137 (Memory Kill)

Cause: Kernel kills container for exceeding memory limits - no warnings, instant death
Impact: Service downtime, lost user sessions, data loss
Fix: Increase memory allocation by 25% minimum

az container show --resource-group mygroup --name mycontainer --query "containers[0].resources"

Reality Check: Apps that use 500MB in dev often need 2GB in production due to caching and dependencies

Image Pull Failures ("Failed to pull image")

Frequency: Most common production failure
Root Causes:

ACI stuck in weird states (60% of cases)
Expired service principal credentials (30% of cases)
Registry connectivity issues (10% of cases)

Fix Priority Order:

Delete and recreate container group (fixes 60% of issues)
Check ACR credential expiration: az acr credential show --name myregistry
Switch to managed identity for credential rotation

Emergency Workaround: Copy image to Docker Hub temporarily

Container Restart Loops (Immediate Death)

Symptoms: Container starts, process exits, ACI restarts, repeat
Debug Method: Override entrypoint with sleep 3600 and exec in
Common Causes:

Missing environment variables (70% of cases)
Database connection failures from Azure (20% of cases)
File permission issues in Docker image (10% of cases)

Container Group Stuck in "Pending"

Cause: Azure resource unavailability
Timeout Threshold: >10 minutes indicates capacity issue
Solutions:

Try different region (East US 2, West Europe, Southeast Asia have best availability)
Reduce resource requirements (4→2 vCPUs, 8→4GB RAM)
Use spot containers with interruption tolerance

Resource Requirements and Limits

Memory Allocation Strategy

Rule: Always request 25% more than app needs
Why: ACI kills containers that exceed limits without graceful handling
Cost Impact: Extra memory costs pennies vs downtime costs

CPU Performance Characteristics

Fractional vCPUs (0.5, 1.5): Poor performance during Azure busy periods
Whole vCPUs (1, 2, 4): Consistent performance for CPU-intensive apps
Spot Containers: 70% cost savings but 30-second eviction notice

Regional Capacity Reality

Small Regions: Limited capacity for large instances (>2 vCPU)
Peak Hours: West Europe frequently unavailable
Reliable Regions: East US 2, West Europe, Southeast Asia

Configuration That Works in Production

Container Startup Optimization

Cold Start Time: 15-60 seconds (can be infinite on bad days)
Image Size Impact: Alpine Linux (5MB) vs Ubuntu (200MB)
Dependency Strategy: Cache in image, don't download at startup

Storage Volume Reality

Azure Files Mounts: Silently fail during network hiccups
Workaround: Check mount exists before writing
Better Solution: Use blob storage with Azure CLI

Multi-Container Group Risks

Problem: One container crash kills entire group
Solution: Separate containers into different groups unless localhost networking required
Exception: Only group containers that absolutely must share localhost

Cost Control and Billing

Pricing by Region (per vCPU-hour)

East US: $0.045
South Central US: $0.043
West Europe: $0.054
Japan East: $0.061

Cost Protection Settings

--restart-policy OnFailure  # Don't restart forever
--max-restart-count 3       # Give up after 3 failures

Confidential Containers

Cost: 3x normal pricing ($0.15 vs $0.045 per vCPU-hour)
Limitations: Breaks privileged operations, hardware access, some Node.js modules
Use Case: Only for truly sensitive data processing

Emergency Debugging Methodology

Step 1: Get Real Error Messages

az container show --resource-group mygroup --name mycontainer --query "containers[0].instanceView"

Look for previousState, currentState, and restartCount

Step 2: Resource Verification

# What you requested
az container show --resource-group mygroup --name mycontainer --query "containers[0].resources.requests"
# What you got
az container show --resource-group mygroup --name mycontainer --query "containers[0].resources.limits"

Step 3: Registry Authentication Test

az acr login --name myregistry
docker pull myregistry.azurecr.io/myapp:latest

Step 4: Container Startup Debug

# Override entrypoint to keep container running
az container create --command-line "sleep 3600"
# Exec in and run app manually
az container exec --exec-command "/bin/bash"

Monitoring and Alerting

Essential Metrics

Restart Count: Alert when >3 in 10 minutes
Memory Usage: Alert when >80% to prevent OOM kills
CPU Usage: Track performance degradation over time

Log Aggregation

Default Retention: 7 days only
Solution: Send to Log Analytics Workspace
Query: Set up alerts for startup failures, not just restarts

Production Patterns That Survive

Always-On Container Design

State Storage: External only (Redis, blob storage)
Database Connections: Connection pooling, not global variables
File Storage: Blob storage, not local filesystem

Scale-to-Zero Anti-Pattern

Problem: 30-90 second cold start kills user experience
Solution: Use Azure Container Apps with min-replicas 1
Valid Use Cases: Batch jobs, CI/CD agents, development environments

Multi-Region Deployment

# Primary region
az container create --location eastus2
# Backup region
az container create --location westus2

Use Traffic Manager for automatic failover during regional issues

Breaking Points and Failure Modes

Memory Limits

Enforcement: Strict, no warnings before kill
Exit Code 137: Kernel memory kill
Prevention: 25% overhead rule

Network Dependencies

VNet Deployment: Requires NAT gateway for outbound connections
Port Mapping: No Docker-style port mapping (container port = exposed port)
Azure Files: Unreliable during network issues

Spot Container Evictions

Warning Time: 30 seconds
Don't Use For: Web apps, databases, long tasks without checkpointing
Good For: CI/CD, log processing, development environments

Critical Warnings

What Microsoft Docs Don't Tell You

ACI availability is unpredictable in smaller regions
Image pull failures are common and often not your fault
Multi-container groups are fragile and should be avoided
Scale-to-zero has terrible user experience for web apps
Azure Files mounts fail silently during network issues

Common Misconceptions

"ACI is reliable for production web apps" - False, use Container Apps instead
"Fractional vCPUs save money" - False, they perform poorly
"Container groups are better than separate containers" - False, unless localhost networking required
"Default monitoring is sufficient" - False, set up proper alerting

Resource Exhaustion Scenarios

Container stuck in restart loop can cost hundreds before detection
Pending containers consume quota without providing service
Memory leaks cause cascading failures in container groups
Regional outages can last hours without alternative regions configured

Useful Links for Further Investigation

Emergency Resources When Everything's Broken

Link	Description
Microsoft's ACI Troubleshooting Guide	The error codes are accurate even if the solutions are generic
ACI Resource Limits and Quotas	Know these limits before you hit them at 3am
Container Group Events and Logs	How to get the real error messages, not the sanitized portal versions
Azure CLI Container Commands Reference	All the az container commands you need for emergency debugging
Docker Best Practices for ACI	Make your images start faster in ACI
Exit Code Reference Guide	What those cryptic exit codes actually mean
ACR Authentication Methods	Service principals, managed identities, and which one to use when
ACR Troubleshooting Guide	Fix "failed to pull image" errors
Azure Managed Identity Setup	Eliminate service principal credential expiration issues
Azure Monitor for Containers	Set up monitoring that catches failures before customers do
Log Analytics Workspace Setup	Store container logs somewhere you can actually query them
Azure Alert Rules for ACI	Alert on restart loops and resource exhaustion
Azure Container Apps vs ACI	When to switch to Container Apps for better reliability
AKS Serverless Options	Virtual nodes give you ACI simplicity with AKS reliability
AWS Fargate Comparison	Sometimes the nuclear option is switching cloud providers

40%

Recommendations combine user behavior, content similarity, research intelligence, and SEO optimization