AWS CodeDeploy - Stop SSHing Into Servers to Deploy Code

CodeDeploy is AWS's answer to the age-old problem of deployments going to shit at the worst possible moment. If you've ever had production go down because someone fucked up a manual deployment, or spent 2 hours at midnight trying to figure out which server didn't get the latest code, CodeDeploy might actually save your sanity.

What It Actually Does

It handles three types of deployments without you having to SSH into anything:

EC2 and On-Premises: Updates regular servers. The agent sits on your boxes, waits for deploy commands, and executes them. Works fine until you forget to install the agent on that one server that breaks everything.

Lambda: Manages serverless function updates with traffic shifting. The canary deployments actually work, unlike when you try to implement them yourself with custom scripts that inevitably break.

ECS: Container deployments with blue/green strategies. Spins up new containers, shifts traffic over, kills the old ones. Takes forever because ECS is slow as hell, but it works.

The Real Benefits (And Pain Points)

Automatic Rollback: When shit breaks, it can automatically rollback. This sounds magical until you realize the health checks can take several minutes to detect failures and your app is already broken. The default timeout is 1 hour per lifecycle event - yes, a bad script can hold your deployment hostage for 60 minutes while customers see errors. In my experience, it takes 5-10 minutes before rollback actually triggers, assuming the health check doesn't time out first.

Scaling: Handles deployments across hundreds of instances simultaneously. This is actually useful when you have a lot of servers. AWS gives you 1,300 concurrent deployments per account, which should handle most scenarios unless you're doing massive multi-region rollouts.

Monitoring: Everything goes through CloudWatch. You get logs, metrics, and events. The logs are usually cryptic as hell when something breaks, but at least they exist.

Cost Reality Check

EC2 deployments are free, which is nice. On-premises instances cost $0.02 each, which adds up fast if you're deploying to thousands of servers. Blue/green deployments will double your instance costs temporarily - something AWS conveniently doesn't mention upfront.

The real cost is the time you'll spend fighting with IAM permissions and debugging why the agent stopped responding for no apparent reason.

So is CodeDeploy worth the pain? That depends on what you're comparing it to. Every deployment tool has its own unique way of ruining your weekend, so let's see how they stack up.

Blue/Green Deployment Reality

Blue/green deployments sound like a miracle until you realize they double your infrastructure costs for however long the deployment takes. But they actually work, which is more than you can say for most DIY deployment strategies.

Here's how it works: CodeDeploy spins up a complete copy of your production environment (the "green" side), deploys your new code there, then shifts traffic from the old environment (the "blue" side). When everything looks good, it kills the old environment.

Traffic Shifting: You get three options that actually make sense:

Canary sends 10% of traffic to the new version for 5 minutes, then switches everyone. Great for catching issues before they kill all your users. The 5-minute wait feels like forever when you're watching metrics, but it's saved my ass more times than I can count.

With Linear, traffic shifts gradually in 10% chunks every 10 minutes. Takes forever but gives you fine-grained control. Use this when you're paranoid about breaking production (which you should be).

Then there's All-at-once - pure YOLO mode that immediately routes all traffic to the new version. Only use this if you're confident your deploy won't break everything, or if you enjoy adrenaline rushes.

The Rollback Promise: CodeDeploy claims it can automatically rollback when shit breaks. In practice, this works about 70% of the time. Last month our e-commerce checkout broke during Black Friday - the health checks took 8 minutes to detect the 500 errors while customers couldn't complete purchases. $30K in lost sales while we watched CodeDeploy slowly figure out something was wrong. The health checks depend on your load balancer configuration, and that 2-10 minute detection window feels like an eternity when your CEO is breathing down your neck.

The AppSpec File Hell

The AppSpec file is your deployment blueprint. It's YAML, which means it will break in creative ways because of indentation issues. Get used to debugging YAML syntax errors at 3am.

Here's the lifecycle phases it runs through:

1. ApplicationStop - Kills your app
2. BeforeInstall - Runs your pre-install scripts
3. Install - Copies files
4. AfterInstall - Post-install scripts (database migrations, etc.)
5. ApplicationStart - Starts your app back up
6. ValidateService - Health checks

Most deployments fail in the ValidateService phase because your health check script has a bug, or the timeout is too short.

Working AppSpec Example:

version: 0.0
os: linux
files:
  - source: /
    destination: /var/www/html
hooks:
  BeforeInstall:
    - location: scripts/install_dependencies.sh
      timeout: 300
      runas: root
  ApplicationStart:
    - location: scripts/start_server.sh
      timeout: 60
      runas: root
  ValidateService:
    - location: scripts/health_check.sh
      timeout: 30

What Actually Breaks in Practice

IAM Permissions: The number one deployment killer. We had a deployment sit in "Pending" for 4 hours because someone rotated the IAM keys without updating the instance profile. Cost us $200 in duplicated blue/green infrastructure while we played "guess which role is missing which permission." The CodeDeploy service role needs different permissions than the EC2 instance role, and the error messages are about as helpful as asking a rock for directions.

Agent Issues: The CodeDeploy agent randomly stops working. Check /var/log/aws/codedeploy-agent/ when deployments just sit there doing nothing. Usually restarting the agent fixes it, but good luck figuring out why it happened.

File Permissions: Your deployment will fail if file permissions are wrong. The agent runs as root but your application might run as a different user. Get familiar with chown and chmod in your AfterInstall hooks.

Network Connectivity: If your instances can't talk to CodeDeploy endpoints, deployments fail silently. Check your security groups and NACLs. The agent needs outbound HTTPS access to AWS APIs.

At this point you're probably thinking "this sounds complicated as hell." You're right. And when things inevitably break (notice I said "when", not "if"), you'll be frantically searching for answers to very specific questions that AWS documentation somehow never covers.