Terraform CLI Production Operations Guide

Critical Configuration Requirements

Production CLI Flags (Essential)

• terraform plan -out=plan.tfplan - Save plans for approval workflows (prevents deployment drift)
• terraform apply plan.tfplan - Apply only approved plans (mandatory for production)
• terraform apply -backup=state.backup - Automatic state backups (prevents data loss)
• NEVER use -auto-approve in production automation - Will cause deployment failures

Performance Optimization Settings

• Container environments: Terraform 1.14+ auto-detects resource limits (fixes CPU exhaustion)
• Manual parallelism control:
  • Slow providers: -parallelism=5 (prevents API throttling)
  • Independent resources: -parallelism=20 (faster deployments)
  • Memory-constrained: Don't exceed GB of RAM in thread count

Critical Failure Recovery Procedures

State Corruption Recovery (3AM Emergency Protocol)

Breaking Point: State corruption causes production outages, prevents deployments
Recovery Sequence (Execute in order):

1. terraform state pull > backup.tfstate - CRITICAL: Never skip this step
2. terraform state list > resources.txt - Document all tracked resources
3. terraform state show <resource> - Inspect corrupted resources
4. terraform state rm <corrupted_resource> - Remove corruption
5. terraform import <resource> <actual_id> - Re-import with correct state
6. terraform plan - Verify no drift (should show no changes)

Failure Mode: Deleting state file destroys infrastructure tracking permanently

Debug Logging for Production Issues

# Separate core vs provider issues
export TF_LOG_CORE=ERROR
export TF_LOG_PROVIDER=DEBUG
export TF_LOG_PATH_MASK="provider_%s.log"

Resource Impact: Debug logs can exceed 50MB, consume significant disk space

Resource Requirements

Time Investment Expectations

• State corruption recovery: 2-4 hours (includes verification)
• Bulk imports with Terraformer: Saves days vs manual (200+ resources)
• Testing framework setup: Previously 20+ minutes teardown, now parallel cleanup
• Container performance issues: Historical 2x-10x slowdown without proper settings

Expertise Prerequisites

• State surgery: Advanced knowledge required (can destroy infrastructure)
• Import operations: Requires existing resource configuration first
• Performance tuning: Understanding of provider rate limits essential

Breaking Points and Failure Modes

Container Runtime Issues (Historical)

Pre-1.14 Behavior: Spawns 20+ threads on 2-CPU containers, causes timeout failures
Current Status: Auto-detection works, manual override available
Workaround: terraform apply -parallelism=2 for constrained environments

Testing Framework Reliability

Historical Issues:

• Random test panics (fixed in recent versions)
• 20-minute sequential teardown (now parallel)
• Variable resolution failures in workspaces (fixed)
  Current State: Functional but still occasional provider-specific failures

Provider-Specific Rate Limits

Provider	Parallelism Limit	Failure Consequence	Workaround
AWS	8 threads max	2+ hour throttling	AWS_MAX_ATTEMPTS=10 AWS_RETRY_MODE=adaptive
Azure	5 threads max	Indefinite throttling	ARM_RATE_LIMIT=15
GCP	12 threads max	Quota exhaustion	Service account optimization

Essential Commands for Production

Emergency Diagnostic Commands

# Immediate drift detection
terraform refresh -no-color > refresh.log 2>&1
terraform plan -no-color > plan.log 2>&1

# Resource dependency analysis
terraform graph | dot -Tpng > dependency.png

# Live state monitoring
watch -n 5 'terraform state list | wc -l'

Expression Testing (Prevents Deployment Failures)

terraform console
> length(var.availability_zones)
> [for zone in var.availability_zones : "${var.region}${zone}"]
> exit

Limitation: Requires valid configuration syntax to start

State Management Operations

# List all tracked resources
terraform state list

# Move resources between addresses
terraform state mv aws_instance.old aws_instance.new

# Remove without destroying
terraform state rm aws_instance.legacy

Terraform Stacks (Multi-Configuration Management)

Operational Benefits

Traditional Pain: Manual dependency management, sequential execution
Stacks Advantage: Automatic dependency resolution, coordinated deployment

Available Operations

terraform stacks -help  # List available subcommands
terraform stacks plan   # Cross-stack planning
terraform stacks apply  # Dependency-aware deployment
terraform stacks status # Cross-stack drift detection

Status: Experimental (use with caution in production)

Testing Framework (Production-Ready)

File-Level Variable Management

# test/main.tftest.hcl
variables {
  environment = "test"
  region     = "us-west-1"
}

run "validate_vpc" {
  command = plan
  
  assert {
    condition     = aws_vpc.main.cidr_block == "10.0.0.0/16"
    error_message = "VPC CIDR must be 10.0.0.0/16 for test environment"
  }
}

CI/CD Integration Requirements

• Terraform version pinning essential for consistent behavior
• Test isolation prevents resource conflicts
• Parallel cleanup reduces CI/CD pipeline time

Import Operations

Legacy Infrastructure Integration

Prerequisites: Write Terraform configuration before importing
Bulk Import Tool: Terraformer for 200+ resource scenarios
Variable Support: Recent versions support workspace variables during import

Import Command Patterns

# Basic import (requires pre-written config)
terraform import aws_instance.web i-1234567890abcdef0

# Import with variables (fixed in recent versions)
terraform import -var="environment=prod" aws_rds_cluster.main cluster-id

Performance Optimization

Refresh Optimization

# Skip refresh for known-good state
terraform apply -refresh=false

# Selective refresh
terraform apply -refresh-only -target=aws_instance.web

Resource Targeting

# Critical resources first
terraform apply -target=module.critical

# Lightweight resources in constrained environments
terraform apply -parallelism=2 -target=module.lightweight_resources

Security and State Management

Remote Backend Migration (Zero Downtime)

1. Configure backend in Terraform configuration
2. Run terraform init (prompts for migration)
3. Verify with terraform plan (no changes expected)
4. Remove local state files: rm terraform.tfstate*
   Critical: Never commit state files to version control

State Locking Resolution

Verification Required: Confirm no active Terraform processes
Backend Check: Examine DynamoDB/backend for stuck locks
Last Resort: terraform force-unlock <LOCK_ID> (corruption risk if process active)

Tool Ecosystem

Essential Production Tools

• TFSwitch: Version management across projects
• Terraformer: Bulk infrastructure import (saves days of manual work)
• TFLint: Static analysis, prevents common errors
• Checkov: Security scanning (750+ policies)
• Driftctl: Infrastructure drift detection
• Infracost: Cost estimation before deployment

Version Management

• Terraform 1.14+: Container performance fixes, testing improvements
• Alpha releases: Experimental features (production risk)
• Version pinning: Essential for team consistency

Common Misconceptions

State File Management

Wrong: "Local state files are fine for small projects"
Reality: Team conflicts cause corruption, remote backend mandatory

Testing Framework

Wrong: "Terraform testing is unreliable"
Reality: Recent versions fixed major issues, now production-ready

Container Performance

Wrong: "Terraform doesn't work well in containers"
Reality: Fixed in 1.14+, previous versions required manual tuning

Import Operations

Wrong: "Import first, then write configuration"
Reality: Configuration must exist before import, prevents drift issues