Velero: Kubernetes Backup & Disaster Recovery - AI Knowledge Base

Executive Summary

Velero is a CNCF-graduated Kubernetes backup tool maintained by VMware Tanzu. Production-proven by Netflix, MongoDB, and Reddit. Current version v1.17.0 (2025) fixes memory leaks but introduces new failure modes. Setup requires 2-3 days due to IAM permission complexity, ongoing maintenance burden moderate.

Core Architecture & Failure Points

Three Critical Components

Velero Server (Controller)

• Function: Watches backup CRDs, communicates with K8s API and storage
• Critical Failure: RBAC/IAM permission mismatches cause silent failures
• Hidden Cost: Requires constant monitoring setup - fails silently by default
• Debugging Time: Hours spent on permission troubleshooting

Velero CLI

• Function: Primary interface for backup/restore operations
• Critical Failure: Commands fail silently, "Completed" status lies
• Required Action: Always run velero backup describe after operations
• Operational Reality: Half of commands require verification to confirm actual success

Node Agent (DaemonSet)

• Function: Handles persistent volume backups via Kopia (replaced Restic in v1.14)
• Critical Failure: Random restarts leave backups stuck in "InProgress"
• Resource Impact: Can OOMKill nodes during large backups without proper limits
• Memory Reality: Massive memory requests that may not be released

Storage Backend Configuration & Costs

AWS S3 (Most Complex Setup)

Setup Time: 2-3 days for IAM permissions
Hidden Requirements: 47+ IAM permissions (official docs incomplete)
Known Issues: GitHub issue #8240 - IRSA roles still broken in recent versions
Storage Cost: ~$25/month per 1TB backup
Breaking Points: Plugin updates frequently break authentication

Critical Permissions Missing from Official Docs:

{
  "Essential_Additions": [
    "s3:AbortMultipartUpload",
    "s3:ListMultipartUploadParts", 
    "ec2:DescribeVolumes",
    "ec2:DescribeSnapshots",
    "ec2:CreateSnapshot",
    "ec2:DeleteSnapshot",
    "ec2:DescribeInstances"
  ]
}

Google Cloud Storage (Least Problematic)

Setup Time: 0.5-1 day
Authentication: Workload Identity cleaner than service accounts
Cost Advantage: Simpler IAM model, readable error messages
Reliability: Most stable authentication mechanism

Azure Blob Storage (Authentication Maze)

Setup Time: 1-2 days navigating identity systems
Complexity: Multiple identity types, subscription dependencies
Breaking Point: Azure's authentication model designed poorly

CSI Volume Snapshots vs File System Backup

Aspect	CSI Snapshots	File System Backup (Kopia)
Speed	Seconds regardless of size	4-8 hours for 500GB
Network Impact	Minimal transfer	Full volume data transfer
Storage Cost	Incremental differences	Full data storage cost
Reliability	Depends on CSI driver quality	Memory leaks fixed but new failure modes
When to Use	Default choice if storage supports	Only when CSI drivers broken
Resource Usage	Minimal	Can OOMKill nodes

Competitive Analysis & Decision Matrix

Solution	Cost Reality	Setup Complexity	Reliability	When to Choose
Velero	Free + debugging time	2-3 days IAM hell	Good with proper monitoring	Multi-cloud, cost-conscious
Kasten K10	$$$ enterprise rates	Sales call required	Actually works	Enterprise budget available
Portworx PX-Backup	$$$ if locked in	Only works with Portworx	Good in ecosystem	Already using Portworx
Native etcd Backup	Free	etcdctl snapshot save	Control plane only	Minimum viable backup

Critical Failure Scenarios & Solutions

"Completed" But Nothing Backed Up

Root Cause: RBAC permissions, storage class incompatibilities, wrong selectors
Detection: Always run velero backup describe backup-name
Prevention: Validate selectors before backup creation

AWS "Access Denied" Loops

Root Cause: Incomplete IAM policies from official docs
Time Cost: 2+ days discovering missing permissions
Solution: Use complete permission set above, check GitHub issues

Stuck "InProgress" Restores

Root Cause: Node agent pod crashed/restarted during operation
Recovery: Delete orphaned restore, restart operation
Prevention: Set appropriate resource limits on node agents

Memory Exhaustion

Root Cause: Kopia memory usage during file system backups
Impact: Node OOMKills, cluster instability
Solution: Configure resource limits, prefer CSI snapshots

Production Implementation Requirements

Resource Planning

Memory: 2-4GB per node agent for large backups
Network: 100GB backup = 100GB transfer with file system method
Time: CSI snapshots (seconds) vs File backup (hours)
Storage: Set retention policies or face budget shock

Monitoring Requirements (Critical)

Why Essential: Velero fails silently by default
Setup Time: 4-8 hours for proper alerting
Key Metrics: Backup success rate, duration, storage usage
Tools: Prometheus + Grafana dashboards mandatory

Testing Protocol (Non-Negotiable)

Quarterly DR Drills: Actually restore in test environment
Validation: Don't trust "Completed" status without verification
Scope: Test both applications and persistent data recovery
Reality Check: Most backup failures discovered during real disasters

When Velero Makes Sense vs Alternatives

Choose Velero When:

• Multi-cloud portability required
• Free solution acceptable with debugging investment
• Team has Kubernetes expertise
• Vendor lock-in unacceptable

Choose Alternatives When:

• Enterprise budget available (Kasten K10)
• Single cloud provider acceptable (AWS/Azure native backup)
• Zero maintenance tolerance
• Immediate reliability required

Critical Warnings & Gotchas

Configuration Traps

Retention Policies: Forgetting these = budget explosion
Secret Restoration: Production secrets to staging = customer emails sent
Plugin Versions: Compatibility breaks between Velero versions
CSI Driver Quality: Many claim snapshot support, few deliver

Cost Surprises

File System Backups: 10x more expensive than CSI snapshots
Network Transfer: Massive bandwidth costs for large volumes
Storage Growth: No retention = exponential cost increase
Hidden AWS Costs: Data transfer fees not mentioned in documentation

Breaking Changes

v1.14 Migration: Restic to Kopia required repository migration
Plugin Updates: Authentication frequently breaks after updates
Kubernetes Versions: Test compatibility before cluster upgrades

Implementation Timeline & Resource Investment

Phase 1: Initial Setup (2-3 days)

• Day 1: IAM/authentication hell
• Day 2: Permission debugging
• Day 3: First successful backup

Phase 2: Production Hardening (1-2 weeks)

• Week 1: Monitoring setup, resource limits
• Week 2: DR testing, failure scenario validation

Phase 3: Operational Maturity (Ongoing)

• Monthly: Review backup success rates
• Quarterly: Full DR drill execution
• As needed: Plugin updates and permission fixes

This knowledge base captures the operational reality of Velero deployment, including hidden costs, time investments, and the specific failure modes that cause production pain. Use this for automated decision-making about whether Velero fits your organization's risk tolerance and resource availability.