Pulumi Deployment Troubleshooting - AI-Optimized Knowledge Base

Critical Configuration Settings

Essential Logging Configuration

• Production Command: pulumi up --logtostderr -v=9 2>&1 | tee deployment.log
• Debug Environment Variables:
  • PULUMI_DEBUG_COMMANDS=true
  • PULUMI_DEBUG_GRPC=true
• Critical Finding: Default Pulumi output is useless - actual errors buried 50-100 lines deep in verbose output
• Search Patterns: Look for lines containing "error", "failed", or cloud provider name

Version Pinning (Production-Critical)

# Pulumi.yaml - REQUIRED for production stability
plugins:
  providers:
    - name: aws
      version: "6.22.2"
    - name: kubernetes
      version: "4.8.1"

• Failure Rate: 90% of "worked yesterday" problems are unpinned provider versions
• Auto-update Risk: Providers auto-update unless explicitly pinned
• Downgrade Command: pulumi plugin install resource aws v5.42.0 --reinstall

Common Failure Modes and Solutions

State Corruption (High Frequency Issue)

Symptoms: "resource creation failed" with no details, deployment stuck "waiting"
Root Causes:

• Manual resource changes in cloud console (60% of cases)
• Partial deployment failures (30%)
• Network/permission issues (10%)

Recovery Process:

1. pulumi stack export --file backup.json (ALWAYS backup first)
2. pulumi refresh (sync state with reality)
3. pulumi import resource-type resource-name actual-cloud-id (manual import required)

Time Investment: 45 minutes to 2 hours depending on resource count

Dependency Violations

Symptom: Resources attempting deletion in wrong order
Immediate Fix: pulumi up --target specific-resource
Force Replacement: pulumi up --replace urn:pulumi:stack::project::aws:rds/instance:Instance::database
Prevention: Add explicit dependsOn properties

Resource Naming Disasters

Critical Warning: Renaming resources in code triggers delete-then-create
Production Impact: Can delete stateful resources (databases, storage)
Safe Rename Process:

1. Export resource: pulumi stack export
2. Import with new name: pulumi import new-type new-name existing-id
3. Remove old resource from state

Resource Requirements and Time Investments

Debugging Time Estimates

Problem Type	Detection Time	Resolution Time	Expertise Required
Provider Version Conflict	5-10 minutes	15-30 minutes	Intermediate
State Corruption	10-20 minutes	45 minutes - 2 hours	Advanced
Dependency Violations	15-30 minutes	30-60 minutes	Intermediate
Resource Import (47 resources)	N/A	3-4 hours	Advanced

Skill Requirements

• Basic: Command-line debugging, reading verbose logs
• Intermediate: State management, provider versions, targeting
• Advanced: Manual imports, circular dependency resolution, production incident response

Production Incident Response

Severity Classification and Response Times

SEV 1 (Production Down):

• Target restoration: 15 minutes maximum
• Bypass Pulumi temporarily - create resources manually
• Import manual fixes later: pulumi import aws:s3/bucket:Bucket emergency-bucket actual-name

SEV 2/3 (Degraded/Minor):

• Full systematic debugging approach
• Root cause analysis required
• Proper state management

Real-World Incident: RDS Deletion

Timeline: 12 minutes to restore service
Root Cause: Resource rename triggered delete-create, creation failed
Data Loss: 10 minutes (recovered from automatic snapshot)
Prevention: Never rename stateful resources without import/export strategy

Critical Warnings and Failure Points

Breaking Points

• UI Performance: Breaks at 1000+ spans, making large transaction debugging impossible
• State File Size: Performance degrades significantly with 500+ resources
• Provider Compatibility: AWS provider 6.0 broke multiple infrastructure patterns
• Regional Limits: Hit quota limits during multi-region deployments

Hidden Costs

• Expertise Requirement: Advanced debugging requires 6+ months Pulumi experience
• Time Investment: Complex state corruption can require full day of engineer time
• Resource Waste: Failed deployments often leave orphaned cloud resources

Common Misconceptions

• Myth: Pulumi handles all dependencies automatically
• Reality: Complex timing dependencies require explicit dependsOn
• Myth: State refresh always fixes drift issues
• Reality: Corrupted state often requires manual imports

"This Will Break If" Scenarios

• Manual changes made in cloud console without Pulumi knowledge
• Provider versions not pinned in production environments
• Renaming resources containing stateful data (databases, storage)
• Deploying during cloud provider maintenance windows
• Running multiple Pulumi operations simultaneously on same stack

Nuclear Options (Last Resort)

When to Use Complete Stack Destruction

• State completely corrupted and refresh/import fails
• Provider versions hopelessly tangled
• Debugging time exceeds 2 hours for single issue
• Multiple cascading failures with unclear root cause

Commands:

# Option 1: Destroy and recreate
pulumi destroy --yes
pulumi up

# Option 2: Force stack removal (loses all state)
pulumi stack rm --force stack-name

Operational Patterns for Success

Prevention Checklist

• Pin all provider versions in Pulumi.yaml
• Set up automated state backups
• Never manually modify cloud resources managed by Pulumi
• Use pulumi preview before all production deployments
• Test resource targeting on individual components
• Implement monitoring for deployment failures

Monitoring and Automation

• State Backup: Daily automated exports to S3/storage
• Deployment Monitoring: Webhook integration to Slack/PagerDuty
• Health Checks: Monitor key resources independent of Pulumi
• Runbook Requirements: Document procedures for RDS deletion recovery, networking failures, certificate expiration

Communication Standards

• Incident Updates: Every 15 minutes during SEV 1
• Customer Communication: Acknowledge quickly, provide regular updates
• Post-Incident: Document root cause, time to resolution, prevention measures

Resource Dependencies and Integration Points

External Tool Integration

• Cloud Provider CLIs: Use for direct resource verification during debugging
• State Management: Pulumi state backends (S3, Azure Blob, GCS)
• Monitoring: Integration with existing infrastructure monitoring
• CI/CD: Pipeline integration requires specific error handling patterns

Community and Support Quality

• High Value: Pulumi Community Slack #help channel - active real-time support
• Moderate Value: GitHub Discussions for complex scenarios
• Variable Quality: Stack Overflow - search existing solutions first
• Official Documentation: Comprehensive but lacks production war stories

This knowledge base represents operational intelligence from production incident response, not theoretical documentation. Use systematic debugging approaches over trial-and-error methods to minimize resolution time and prevent cascading failures.