Velero - Save Your Ass When Kubernetes Implodes

Velero is the backup tool you install after experiencing your first major Kubernetes disaster. Originally called Heptio Ark, it's now maintained by VMware Tanzu and has graduated from the CNCF, which means it's stable enough that you won't get fired for using it.

v1.17.0 dropped in 2025 with Windows support and micro-service architecture for fs-backup. The big deal is they fixed the memory hogging issues - node-agent pods used to request massive memory and hold it forever. Now they actually release it when done backing up your stuff.

Companies like Netflix, MongoDB, and Reddit run this in production, which tells you it's battle-tested enough for your workloads.

The Three Parts That Actually Matter

Velero has three components that you'll spend way too much time debugging:

Velero Server: The controller that lives in your cluster and watches for backup CRDs. When it works, it talks to the K8s API and your storage backend. When it doesn't work, you'll spend hours checking RBAC permissions and IAM policies. The server handles backup schedules but won't tell you when they fail unless you set up monitoring.

Velero CLI: Your interface for everything. velero backup create, velero schedule create, velero restore create - sounds simple until you realize half the commands fail silently. The CLI has validation but it's optimistic. Always check velero backup describe after creating anything because "Completed" doesn't mean what you think it means.

Node Agent: DaemonSet that does the heavy lifting for persistent volume backups when CSI snapshots aren't available. Since v1.14 it uses Kopia instead of Restic, which fixed the memory leaks but introduced new failure modes. Pro tip: these pods will randomly restart and leave your backups stuck in "InProgress" status.

When You Actually Need This Thing

Velero solves three problems that will ruin your day:

Disaster Recovery: When someone accidentally deletes prod or your cloud provider has an outage. Netflix uses this because they learned the hard way that shit happens. Recovery time is anywhere from 10 minutes if you're lucky to 6 hours if your persistent volumes are massive. Don't ask me how I know.

Cluster Migration: Moving workloads between clusters without wanting to rebuild everything from scratch. The official migration docs make it sound easy, but you'll spend days fixing storage classes, ingress annotations, and whatever custom resource definitions broke between environments. Test this multiple times before doing it in prod.

Environment Replication: Copying prod to staging so developers can debug with real data instead of made-up test fixtures. Works great until you restore production secrets to staging and suddenly your test environment is sending real emails to customers. Always sanitize before restoring.

Storage Options (And Where They'll Bite You)

Velero supports multiple storage backends, each with its own special way of frustrating you:

AWS S3: The AWS plugin works great once you survive the IAM permission hell. You'll need about 47 different permissions, and the documentation lies about half of them. GitHub issue #8240 from September 2024 shows the plugin still fucks up IRSA roles. Budget 2 days for getting the permissions right, then another day when it breaks after an AWS update.

Google Cloud Storage: The GCP plugin is honestly the least painful to set up. Workload Identity is cleaner than dealing with service account keys, and Google's IAM model actually makes sense. Still fails sometimes but at least the error messages are readable.

Azure Blob Storage: The Azure plugin works with managed disks and blob storage. Managed identity is nice when it works, but Azure's authentication model is a maze. Expect to spend time figuring out which identity goes where.

CSI Volume Snapshots: Uses your CSI driver to take volume snapshots instead of copying files. Fast and efficient when it works, but many CSI drivers have bugs. AWS EBS snapshots are reliable, others are hit-or-miss. Always test restoring from snapshots - taking them is the easy part.

S3-Compatible Storage: MinIO and Ceph work for on-premises setups. MinIO is solid, Ceph will make you question your life choices. Good for air-gapped environments where cloud storage isn't an option.

What You Actually Need

Velero needs Kubernetes v1.20+ - basically any modern cluster. v1.17.0 is current and works with K8s up to 1.31. The hard part isn't the version compatibility, it's the fucking IAM permissions that'll make you question your career choices.

Step-by-Step Installation Hell

Step 1: Get the CLI (This Part Actually Works)

## Download v1.17.0 - check GitHub for latest
wget https://github.com/vmware-tanzu/velero/releases/download/v1.17.0/velero-v1.17.0-linux-amd64.tar.gz
tar -xvf velero-v1.17.0-linux-amd64.tar.gz
sudo mv velero-v1.17.0-linux-amd64/velero /usr/local/bin/

Step 2: AWS Setup (Where Dreams Go to Die)

Create your S3 bucket first - this part is easy:

aws s3 mb s3://your-velero-bucket-name-here

Now comes the IAM clusterfuck. The official AWS docs give you a policy that's missing half the permissions you actually need. You'll discover the missing ones when backups fail silently. Save yourself hours and use this:

{
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject", 
                "s3:PutObject",
                "s3:AbortMultipartUpload",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": "arn:aws:s3:::your-bucket/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::your-bucket"
        }
    ]
}

Add EBS snapshot permissions if you want volume snapshots (and you do):

{
    "Effect": "Allow",
    "Action": [
        "ec2:DescribeVolumes",
        "ec2:DescribeSnapshots",
        "ec2:CreateSnapshot",
        "ec2:DeleteSnapshot",
        "ec2:DescribeInstances"
    ],
    "Resource": "*"
}

Step 3: Install and Pray

velero install \
    --provider aws \
    --plugins velero/velero-plugin-for-aws:v1.11.0 \
    --bucket your-velero-bucket-name-here \
    --backup-location-config region=us-east-1 \
    --snapshot-location-config region=us-east-1 \
    --secret-file ./credentials-velero

This will probably fail the first time with some cryptic error about credentials. Check kubectl logs -n velero deployment/velero to see what's actually broken.

The Parts That Break Most Often

Backup Storage Locations (BSL): This is where your backups actually go - an S3 bucket or compatible object storage. When backups fail, it's usually because the BSL can't authenticate. Check kubectl get backupstoragelocation -n velero and look for "Available: false". Multiple backup storage locations are possible but add complexity you probably don't need.

Volume Snapshot Locations (VSL): Where volume snapshots live, usually in the same region as your cluster to avoid data transfer costs. When snapshots fail, check if your storage provider supports CSI snapshots properly. Many don't, despite claiming they do.

Custom Resource Definitions: Velero adds these CRDs that you'll debug constantly:

• `Backup`: Your backup job. Check .status.phase - "Completed" with warnings usually means it didn't back up what you think it did
• `Restore`: Restore job. "InProgress" stuck forever means check the logs with velero restore logs
• `Schedule`: Cron-based backup scheduling. Silent failures are common - set up monitoring
• `BackupStorageLocation` and `VolumeSnapshotLocation`: Configuration CRDs that break when IAM policies change

Two Ways to Back Up Volumes (One Works, One Doesn't)

CSI Volume Snapshots: Uses your cloud provider's native snapshot API. Works with AWS EBS, GCP Persistent Disks, and Azure Disks. Takes seconds regardless of volume size and costs way less than file backup. This is what you want to use unless your storage provider's CSI driver is broken (many are).

File System Backup: Uses Kopia to copy actual files from your volumes. Replaced Restic in v1.14 which fixed the memory leaks but introduced new ways for backups to fail. Takes forever, uses tons of network bandwidth, and costs more in storage. Use this only when CSI snapshots don't work with your storage setup.

Production Reality Check

Resource Usage: File system backups will eat your cluster resources alive. The node agent pods can request massive amounts of memory during Kopia operations. Set appropriate resource limits or watch your nodes get OOMKilled during backup operations.

Network Bandwidth: File system backups upload your entire volume data to object storage. A 100GB database backup will transfer 100GB over the network and take 2 hours on a decent connection. CSI snapshots transfer almost nothing. Plan accordingly.

Storage Costs: Forgot to set retention policies? Your first AWS bill will be a surprise. File system backups cost more because they store actual data. CSI snapshots use incremental differences, so they're cheaper. Set retention or go broke.