HashiCorp Packer - Automated Machine Image Builder

What is HashiCorp Packer and Why Use It?

Packer builds machine images so you don't have to spend your weekends manually configuring servers. Latest version Packer 1.11.0 dropped July 2024, but honestly, the plugin bugs still make you want to throw your keyboard through a window half the time.

If you're tired of deployment surprises and "works on my machine" disasters, Packer might save your sanity. Here's the problem it actually solves.

The "Works On My Machine" Problem

Ever spent 3 hours debugging why your app works perfectly on your MacBook but dies instantly in production? Yeah, that's the "works on my machine" curse. Traditional deployments involve spinning up a fresh server and installing everything from scratch, hoping nothing breaks. Spoiler alert: something always breaks.

Packer solves this by baking everything into the golden image beforehand. Your app, dependencies, configs, security patches - all pre-installed and tested. When you deploy, you're just booting an image that already works.

How Packer Actually Works

Packer Workflow Diagram

Packer has three parts that actually do stuff:

Builders: Where to build your images (AWS, Azure, Docker, VMware)
Provisioners: Scripts that install your app (shell scripts work fine, Ansible if you're feeling fancy, Chef if you hate yourself)
Post-processors: What happens after - usually uploading to a registry or compressing the result

The process looks simple on paper: write a template in HCL2 (HashiCorp's answer to "YAML isn't complicated enough"), run packer build, and sacrifice a goat. Packer spins up a VM, runs your provisioning scripts, shuts it down, and saves the image. Sounds easy until your scripts fail because they expect your laptop's specific PATH or that random symlink you created months ago.

The Good and Bad News

Good news: Packer can build images for multiple platforms simultaneously. I cut our deployment time from 2 hours of manual server setup to 5-minute image boots. Worth the initial pain of getting Packer templates working.

Bad news: Build times vary wildly. Basic Linux AMIs take 5-10 minutes. Windows images with Visual Studio? Plan for 45 minutes and pray nothing times out. Docker builds are fast (2-5 minutes) unless you're installing every npm package known to humanity.

AWS Logo Azure Logo

Packer runs on Linux, macOS, and Windows, though Windows support feels like they only tested it once. The binary is tiny (~100MB) but builds will max out your CPU and network like it's 1999.

Packer vs Alternative Image Building Solutions

Feature	HashiCorp Packer	AWS EC2 Image Builder	Azure Image Builder	Docker Build	Red Hat Image Builder
Multi-Cloud Support	✅ 50+ platforms	❌ AWS only	❌ Azure only	✅ Multi-platform	❌ RHEL/CentOS only
Configuration Language	HCL2, JSON	YAML, CLI	JSON, ARM templates	Dockerfile	TOML
Parallel Builds	✅ Yes	❌ Sequential	❌ Sequential	✅ Multi-stage	❌ Sequential
Licensing	Business Source License	AWS Managed	Azure Managed	Open Source	Open Source
Learning Curve	Moderate	Low (AWS native)	Low (Azure native)	Low	Moderate
Infrastructure Required	Local/Remote	AWS managed	Azure managed	Local/Remote	Local
Cost Model	Free (usage charges apply)	Pay per build + storage	Pay per build + storage	Free	Free
Image Formats	AMI, VHD, VMDK, OVF, Docker	AMI only	VHD, Managed Images	Container images	ISO, QCOW2, VMDK
Provisioner Support	Shell, Ansible, Chef, Puppet	Systems Manager, Scripts	PowerShell, Shell	RUN commands	Built-in tools
Template Reusability	✅ Excellent	❌ Limited	❌ Limited	✅ Good	❌ Limited
Community Plugins	100+ plugins	AWS marketplace	Azure extensions	Docker Hub	Limited

Real Production Experience and Pain Points

Now for the real shit nobody puts in the documentation - what happens when you actually run Packer in production. Spoiler: you'll be debugging failed builds at 2 AM more than you'd like.

What Actually Happens in Production

HCP Packer is HashiCorp's hosted image registry. It's nice when it works, but costs add up fast if you're building many images. We pay ~$200/month for image storage and metadata.

The "golden image pipeline" sounds amazing until reality hits. You'll burn through weeks figuring out why scripts that work perfectly on your laptop suddenly shit themselves in Packer. Here's what nobody warns you about:

Timing matters: Your scripts might run before network interfaces are up
Environment differs: Packer builds run as root, your local tests probably don't
Timeouts kill everything: Windows builds timeout constantly (default 5min is a joke)
Cleanup is manual: Failed builds leave zombie EC2 instances running up your bill

Real Deployment Scenarios

Multi-Cloud Hell: We build the same image for AWS and Azure because management loves vendor-agnostic buzzwords. The configs need tweaking because Azure does networking differently (shocking, I know), but it beats maintaining two completely separate pipelines. Test on both platforms every single time - subtle differences will bite you in the ass at the worst possible moment.

CI/CD Integration: Packer in GitHub Actions works fine until AWS decides to have a bad day. Your builds will randomly fail because t3.micro doesn't exist in us-east-1c today, or networking hiccups during provisioning, or you hit AWS API rate limits that nobody documented properly. Always retry failed builds - about half the time it's just AWS being AWS.

Pro tip: Always set instance_type = "t3.small" for anything serious. t3.micro runs out of memory during Docker builds and you'll waste hours debugging OOM kills.

Disaster Recovery: Banks love Packer because they can rebuild identical environments across regions. Reality: you'll discover your AMI permissions don't work cross-account when disaster actually strikes. Test your DR process, don't just assume it works.

Lessons From 3 Years of Production Packer

Packer HCL Template Structure

Template Structure: Start simple, add complexity later. Our hierarchy:

base.pkr.hcl - OS + security patches
app-base.pkr.hcl - builds on base, adds runtime (JVM, Node, whatever)
service.pkr.hcl - builds on app-base, adds your actual application

Docker Logo

Security Scanning: We integrated Trivy into builds. Scans add 2-3 minutes but catch CVEs before production. Better than explaining to security why your container has 847 critical vulnerabilities (yes, 847 - I actually counted - most from a Node.js 12 base image that was ancient when dinosaurs roamed the earth).

Version Everything: Store templates in Git with your app code. Tag releases. When something breaks (and it will), you can rollback to the last working image. I learned this after spending 6 hours debugging why our "quick config change" broke login for 50,000 users.

Another gotcha nobody warns you about: Security groups must allow port 22 inbound or your builds will timeout trying to SSH in. Obvious in retrospect, not so much at 2 AM when you're wondering why every build fails with "connection refused".

Questions Everyone Asks (And The Real Answers)

What's the difference between Packer and Docker?

Docker builds containers that share the host kernel. Packer builds full VM images with their own kernel. Think of Docker as "put your app in a box" and Packer as "build an entire server with your app pre-installed."Packer can build Docker images too, which confuses everyone initially. Use Docker for microservices, Packer for golden AMIs that boot in 30 seconds instead of taking 20 minutes to provision.

Does Packer replace Ansible/Chef?

No, Packer uses Ansible/Chef/Puppet as provisioners during image builds. The difference is timing:

Ansible normally: Runs after you boot servers (slow, can fail)
Ansible with Packer: Runs during image creation (fails fast, debug locally)

I stick with shell scripts in Packer because they're predictable. Ansible playbooks that run flawlessly on your laptop turn into steaming piles of failure in Packer builds. Ansible makes assumptions about the environment (like systemd being ready, or certain users existing) that Packer's minimal build environment doesn't meet. Shell scripts are dumber but at least they fail in predictable ways.

How much does Packer cost?

Packer binary is free under BSL license. You pay for AWS/Azure compute while building (usually $2-10 per build depending on instance size and build time).

HCP Packer costs $0.35/image/month for the registry. Sounds cheap until you have 200+ images across environments. We switched to storing metadata in Git after getting a $400/month HCP bill.

Can Packer build images for multiple clouds simultaneously?

Yeah, and it's actually one of the few things Packer does really well. One template can build for AWS, Azure, Docker, VMware

whatever you need. Packer runs these builds in parallel so you're not sitting around waiting for sequential builds to finish. This feature alone makes up for a lot of Packer's other quirks.

What happens when Packer builds fail?

When Packer fails, it usually cleans up after itself. Usually. Sometimes you'll find zombie EC2 instances running up your bill. Always double-check your AWS console after failed builds.

Use packer build -debug template.pkr.hcl to pause at each step and SSH into the build instance. This is the only way to debug why your provisioning scripts are failing. Pro tip: check /tmp/packer-provisioner-shell-* for script output.

Does Packer work with Windows?

Yes, but Windows builds take longer than a CVS checkout over dial-up. Plan for 45+ minutes even for basic images. Packer connects via WinRM (port 5985) which you'll need to enable in your base image.

Packer Verification

Windows builds timeout constantly because the default 5-minute limit is a joke. Set ssh_timeout = "10m" for Windows builds or you'll timeout during sysprep. I've seen Windows updates take 30 minutes during image builds - and they WILL start updating during your build whether you want them to or not.

How do I handle secrets in Packer?

Never hardcode secrets in templates - they end up in logs and Git history. Options:

Environment variables: export DB_PASSWORD=secret && packer build
AWS Systems Manager: {{ssm "/prod/db/password"}}
HashiCorp Vault: if you're already in their ecosystem
User variables: Pass secrets at build time with -var

I use AWS Systems Manager Parameter Store because it's simple and already encrypted.

Can I build on top of existing images?

Yes, and you should. Starting from existing AMIs/images is faster than building from scratch. Common pattern:

Start with official Ubuntu/Amazon Linux AMI
Add your app-specific configs
Save as your golden image

Builds are incremental - if you change one line in a provisioning script, you don't rebuild the entire OS. Just make sure your base images get security updates regularly.

Why do my builds keep failing randomly?

Welcome to the wonderful world of Packer, where builds fail for reasons that would make a grown developer cry. Enjoy these greatest hits of random failures:

Instance type not available: AWS runs out of t3.micro in us-east-1c again
Network timeouts: Your provisioning script tries to download something and AWS networking hiccups
Permissions: Your IAM role can launch instances but not create AMIs
Disk space: You're installing too much shit and running out of space
The dumb thing to check first: Is your VPC subnet actually public? Private subnets can't download packages from the internet
Nuclear option: Delete everything and try again - works about 30% of the time

Always retry failed builds once before panicking - about half the time it's just AWS or Azure having infrastructure hiccups. If it fails twice with the same error, congratulations, it's actually your problem now.

Time estimates based on reality: 5 minutes if you're lucky, 2 hours if you're not. Plan accordingly.

How do I use Packer in CI/CD?

Packer has a CLI so it works with any CI system. In GitHub Actions:

- name: Build AMI
  run: packer build -var "version=${{ github.sha }}" ami.pkr.hcl

Common gotchas: AWS credentials, build timeouts in CI, and parallel builds hitting resource limits. I build images on every main branch push, tag them with git SHA, and use the latest successful build for deployments.

Real gotcha from production: IAM permissions are a nightmare - you need ec2:* and basically everything else. Start with PowerUser and tighten from there.

Should I use the BSL licensed version?

HashiCorp changed to Business Source License which allows free use unless you're building a competing product. For normal use (building internal infrastructure images), you're fine. If you're building a SaaS that competes with HCP Packer, you might need the commercial license.

Packer vs AWS Image Builder vs Docker?

Packer: Multi-cloud, flexible, steep learning curve
AWS Image Builder: AWS only, GUI-driven, limited customization
Docker: Fast builds, container-only, different use case

Use Packer if you need multi-cloud or complex provisioning. Use AWS Image Builder if you only do AWS and want something simple. Use Docker for containerized apps.