Terraform State Corruption: Recovery & Prevention Guide

Critical Failure Modes

Primary Causes of State Corruption

• Network interruption during apply: Most common cause. WiFi/network failure during state upload to S3 results in partial JSON file
• Concurrent executions: Two users running terraform apply simultaneously without proper locking
• Disk space exhaustion: Docker volumes at 100% capacity truncate state files to 0 bytes
• Manual state editing: Single typo in JSON breaks entire state file
• Provider version incompatibility: AWS provider 5.x to 6.x migration broke state files with schema changes

Severity Assessment

• Level 1 - JSON Syntax Error: 30-minute fix with backups, catastrophic without
• Level 2 - Partial Corruption: 1 day of importing missing resources
• Level 3 - Total State Loss: 2-3 days for medium environments (50-500 resources), entire weekend cancelled for large deployments

Immediate Symptoms

• terraform plan fails with "Error: invalid character" JSON errors
• terraform state list returns empty despite running resources
• Resources show as "new" when they already exist
• Commands hang indefinitely with lock errors

Recovery Procedures

Option 1: Backup Restoration (20-60 minutes)

Local Backup Recovery

# Verify backup integrity
cat terraform.tfstate.backup | jq . > /dev/null

# Restore if valid
cp terraform.tfstate.backup terraform.tfstate
terraform plan  # Verify functionality

S3 Versioning Recovery

# List all state file versions
aws s3api list-object-versions \
  --bucket terraform-state-bucket \
  --prefix prod/terraform.tfstate

# Download pre-corruption version
aws s3api get-object \
  --bucket terraform-state-bucket \
  --key prod/terraform.tfstate \
  --version-id VERSION_ID \
  terraform.tfstate.backup

# Restore to current state
terraform state push terraform.tfstate.backup

Option 2: Manual Import (1-3 days)

Resource Discovery

# AWS inventory
aws ec2 describe-instances --output table
aws rds describe-db-instances --output table
aws s3api list-buckets

# Azure inventory  
az resource list --output table

# GCP inventory
gcloud compute instances list
gcloud sql instances list

Import Process (Dependency Order)

1. VPCs and networking - everything depends on these
2. Security groups and IAM roles
3. EC2 instances, RDS databases
4. Load balancers and DNS
5. Application-specific resources

# Create minimal resource configuration
resource "aws_instance" "web" {
  ami           = "ami-12345678"
  instance_type = "t3.micro"
  # Refine after import
}

# Import actual resource
terraform import aws_instance.web i-1234567890abcdef0

# Verify and fix configuration
terraform plan

Option 3: Emergency Temporary State

For critical production issues requiring immediate deployment:

# Initialize new state
terraform init

# Import only critical resources
terraform import aws_instance.prod_web i-critical-instance-id
terraform import aws_security_group.prod sg-whatever

# Create minimal viable configuration
# Fix comprehensive state later

Prevention Configuration

Remote State Setup (Required)

S3 + DynamoDB Backend

terraform {
  backend "s3" {
    bucket         = "company-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "us-west-2"
    encrypt        = true
    dynamodb_table = "terraform-state-locks"
    versioning     = true  # Critical for recovery
  }
}

Infrastructure Setup

# Enable S3 versioning (saves your career)
aws s3api put-bucket-versioning \
  --bucket terraform-state-bucket \
  --versioning-configuration Status=Enabled

# Create DynamoDB lock table
aws dynamodb create-table \
  --table-name terraform-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

Automated Backup System

Daily Backup Script

#!/bin/bash
# Run daily via cron: 0 2 * * *

DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_BUCKET="terraform-backups"

for env in prod staging dev; do
  cd /path/to/${env}/terraform
  
  # Pull and validate state
  terraform state pull > "backup-${env}-${DATE}.json"
  
  if jq . "backup-${env}-${DATE}.json" > /dev/null; then
    aws s3 cp "backup-${env}-${DATE}.json" "s3://${BACKUP_BUCKET}/${env}/"
    echo "${env} backup successful"
    rm "backup-${env}-${DATE}.json"
  else
    echo "ERROR: ${env} backup corrupted"
  fi
done

State Health Monitoring

#!/bin/bash
# Monitor state file integrity

CURRENT_SIZE=$(terraform state pull | wc -c)
if [ $CURRENT_SIZE -lt 1000 ]; then
  echo "WARNING: State file suspiciously small (${CURRENT_SIZE} bytes)"
fi

RESOURCE_COUNT=$(terraform state list | wc -l)
if [ $RESOURCE_COUNT -eq 0 ]; then
  echo "ERROR: State file contains no resources"
fi

Team Workflow Configuration

GitOps CI/CD Pipeline

# .github/workflows/terraform.yml
name: Terraform CI/CD
on:
  pull_request:
    paths: ['terraform/**']
  push:
    branches: [main]

jobs:
  terraform:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.5.7
      
      - name: Terraform Init
        run: terraform init
        
      - name: Terraform Plan
        if: github.event_name == 'pull_request'
        run: terraform plan -no-color
        
      - name: Terraform Apply
        if: github.ref == 'refs/heads/main'
        run: terraform apply -auto-approve

Environment Separation

terraform/
├── prod/       # Separate S3 keys
│   ├── main.tf
│   └── backend.tf (key = "prod/terraform.tfstate")
├── staging/
│   ├── main.tf  
│   └── backend.tf (key = "staging/terraform.tfstate")
└── dev/
    ├── main.tf
    └── backend.tf (key = "dev/terraform.tfstate")

Resource Requirements & Time Investment

Recovery Time Estimates

• Small environment (<50 resources)

  • Backup restore: 20 minutes
  • Manual import: 4-6 hours
  • Complete rebuild: 8+ hours

• Medium environment (50-500 resources)

  • Backup restore: 30-60 minutes
  • Manual import: 1-2 days
  • Complete rebuild: 3-5 days

• Large environment (500+ resources)

  • Backup restore: 1-2 hours
  • Manual import: 3-7 days
  • Complete rebuild: 1-2 weeks

Prevention Costs

• S3 + DynamoDB backend: $10-50/month depending on state size
• Daily backup storage: $5-20/month
• CI/CD platform: $0 (GitHub Actions) to $50+/month (commercial)

Human Resource Impact

• State corruption incident: 1-3 engineers blocked for days
• Prevention setup: 1 engineer for 1-2 days initial configuration
• Maintenance: 2-4 hours/month monitoring and updates

Common Failure Scenarios

Lock File Issues

Symptom: "Error acquiring the state lock" preventing all operations
Cause: Process killed during apply, leaving orphaned lock
Resolution:

# Identify lock holder
aws dynamodb scan --table-name terraform-locks

# Force unlock (ensure no active operations first)
terraform force-unlock LOCK_ID

Provider Migration Failures

Symptom: Resources show as completely different types after provider upgrade
Cause: Schema changes between major provider versions
Prevention: Test upgrades in non-production first, maintain provider version constraints

Partial State Corruption

Symptom: Some resources missing from state, others intact
Cause: Interrupted writes, filesystem issues
Resolution: Selective import of missing resources rather than full rebuild

Critical Configuration Settings

Required S3 Bucket Configuration

• Versioning: Enabled (provides automatic backups)
• Encryption: AES256 or KMS (security compliance)
• Public access: Blocked (security requirement)
• Lifecycle policy: Retain 30+ versions

Required DynamoDB Table Settings

• Partition key: LockID (String type)
• Billing: Pay-per-request (cost optimization)
• Point-in-time recovery: Enabled (additional safety)

Terraform Configuration Requirements

# Minimum backend configuration
terraform {
  required_version = ">= 1.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"  # Pin major version
    }
  }
  
  backend "s3" {
    encrypt        = true
    dynamodb_table = "terraform-locks"
    # Other settings environment-specific
  }
}

Emergency Procedures

Production State Corruption Response

1. Immediate: Stop all Terraform operations across team
2. Assessment: Determine corruption scope (partial vs total)
3. Communication: Notify stakeholders of deployment freeze
4. Recovery: Attempt backup restoration first, import as fallback
5. Verification: Extensive testing before resuming operations
6. Post-incident: Review prevention measures, update runbooks

Disaster Recovery Checklist

• State backups accessible and tested monthly
• Recovery procedures documented and practiced
• Team trained on emergency procedures
• Escalation paths defined for different severity levels
• Rollback procedures prepared for critical changes

Operational Intelligence

What Official Documentation Doesn't Tell You

• Workspace vs separate files: Workspaces are development features, not environment separation
• State locking limitations: DynamoDB locks don't prevent all race conditions
• Import complexity: Complex resources require extensive configuration matching
• Backup timing: S3 versioning alone insufficient for high-frequency changes

Community Wisdom

• Terraformer tool: Generates configs but requires significant cleanup
• Import order matters: VPC resources first, applications last
• Large state performance: Files >100MB become unwieldy, split recommended
• Provider stability: Major version upgrades often break state compatibility

Breaking Points and Limitations

• UI performance: Terraform Cloud becomes unusable with 1000+ resources
• Apply duration: State files >50MB cause 10+ minute plan times
• Import limitations: Some resources impossible to import accurately
• Lock timeout: Default 20-minute timeout insufficient for large applies