Pulumi Kubernetes Helm GitOps Production Implementation Guide

Executive Summary

This is a comprehensive production implementation guide for integrating Pulumi, Kubernetes, Helm, and GitOps workflows. The content provides real-world operational intelligence gathered from 18 months of production experience, including failure scenarios, resource requirements, and cost implications.

Critical Resource Requirements

Minimum Viable Production Setup

• Monthly AWS Cost: $1,200-$1,500 (3 environments with monitoring)
• Setup Time: 6 months to production-ready
• Team Investment: $100K+ for proper migration
• Minimum Cluster: 3x t3.medium nodes ($200/month base cost)

AWS Cost Breakdown

Component	Monthly Cost	Notes
EKS Control Plane	$73/cluster	Non-negotiable AWS charge
Worker Nodes (3x t3.medium)	$67	Minimum for stability
LoadBalancers (5 services)	$90	$18/month each
NAT Gateway	$45	Required for outbound internet
Data Transfer	$20-40	Cross-AZ charges
EBS Volumes	$15-30	Container storage
Total Minimum	$310-345	Testing environment only

Resource Specifications That Actually Work

ArgoCD Production Resource Limits

controller:
  resources:
    requests:
      memory: "2Gi"     # Will OOMKill with less
      cpu: "1000m"      
    limits:
      memory: "4Gi"     # Scales with application count
      cpu: "2000m"      # Needs bursting for large syncs

server:
  resources:
    requests:
      memory: "512Mi"   # UI is memory hungry
      cpu: "250m"
    limits:
      memory: "1Gi"     # UI has memory leaks
      cpu: "500m"

repoServer:
  resources:
    requests:
      memory: "512Mi"
      cpu: "250m"
    limits:
      memory: "1Gi"
      cpu: "500m"

EKS Node Configuration

const nodeConfig = {
    dev: { 
        instanceType: "t3.small", 
        nodeCount: 2, 
        maxNodes: 3,
        cost: "$150/month"
    },
    staging: { 
        instanceType: "t3.medium", 
        nodeCount: 2, 
        maxNodes: 4,
        cost: "$300/month"
    }, 
    prod: { 
        instanceType: "t3.large", 
        nodeCount: 3, 
        maxNodes: 10,
        cost: "$800+/month"
    }
};

Critical Failure Modes and Solutions

High-Frequency Issues (Weekly Occurrence)

1. ArgoCD Application Stuck "Progressing"

Frequency: Weekly
Duration: 5-10 minutes to 6+ hours
Root Causes:

• ArgoCD controller OOMKilled
• RBAC permission issues
• Kubernetes API server timeout
• ArgoCD internal state corruption

Solutions (in order of success rate):

# 90% success rate - restart controller
kubectl rollout restart deployment argocd-application-controller -n argocd

# If that fails - nuclear option
kubectl delete application your-app -n argocd
# Wait 2 minutes, then reapply YAML

2. Pulumi State Lock/Corruption

Frequency: Monthly
Impact: Blocks all infrastructure changes
Prevention: Never run pulumi up manually with GitOps

Recovery Process:

# 1. Try to cancel operations
pulumi cancel

# 2. Clear lock (dangerous but necessary)
pulumi state delete-lock <lock-id>

# 3. Nuclear option - export/reimport state
pulumi stack export > stack-backup.json
pulumi stack rm --force
pulumi stack init <same-name>
pulumi stack import < stack-backup.json

3. Helm Dependency Resolution Failures

Error: "repository not found"
Frequency: Weekly
Root Cause: Helm caching system is broken by design

Fix:

# Clear Helm cache (fixes 60% of issues)
helm repo update
helm dependency update charts/your-app/

# Nuclear option
rm -rf ~/.cache/helm/
rm -rf charts/your-app/charts/
helm dependency build charts/your-app/

Medium-Frequency Issues (Monthly Occurrence)

1. AWS Networking Failures

Issue: VPC CNI runs out of IP addresses despite available subnet space
Impact: New pods cannot schedule
Solution: Use larger instance types or custom CNI configuration

2. LoadBalancer IP Assignment Failures

Issue: AWS Load Balancer Controller fails silently
Duration: 3-8 minutes when working, infinite when broken
Detection: kubectl get svc --watch shows <pending> forever

3. Container Image Pull Failures

Cause: ECR authentication expiration or IAM role misconfiguration
Impact: Applications stuck in ImagePullBackOff
Debug: kubectl describe pod shows specific error

Production Implementation Patterns

Environment Separation Strategy

DO: Separate clusters per environment
DON'T: Use namespace isolation in single cluster
Reason: Resource contention causes production incidents

GitOps Promotion Workflow

# Development - full automation
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

# Staging - automated with manual promotion gates
spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: false

# Production - manual deployment only
spec:
  syncPolicy: {}  # No automation

Multi-Region Disaster Recovery

Reality: Full multi-region doubles AWS costs
Alternative: Fast recovery strategy

• RTO: 4-6 hours (rebuild from scratch)
• RPO: 5 minutes (database backups)
• Cost: 20% of dual-region approach

Security Implementation

External Secrets Management

Recommended: External Secrets Operator with AWS Secrets Manager
Cost: $0.40/month per secret
Alternative Evaluation:

• SOPS: Demo-ready, operations nightmare
• Vault: Enterprise-grade, $150K+/year licensing
• Sealed Secrets: Works but limited features

Production Security Configuration

# External Secrets Operator pattern
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-west-2
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa

Monitoring and Observability

Critical Metrics (The Only 10 That Matter)

1. ArgoCD Controller Up/Down
2. Pulumi Stack Success/Failure Rate
3. Helm Release Success/Failure Rate
4. Kubernetes API Server Availability
5. Node Ready Status
6. Pod Crash Loop Detection
7. Resource Usage (CPU/Memory/Disk)
8. Application Response Times (user-facing only)
9. Recent Deployment Timeline
10. LoadBalancer Health Status

Production Dashboard Requirements

Rule: If on-call engineer can't understand in 30 seconds at 3AM, it's useless
Panels: Maximum 6 panels

• Cluster health (green/red status)
• ArgoCD sync failures (red alerts only)
• Pod status by namespace
• Resource utilization
• User-facing service response times
• Recent deployment history

Performance Optimization

Cluster Autoscaling Configuration

nodeGroups:
  general:
    instanceTypes: ["t3.medium", "t3.large"]
    minSize: 2
    maxSize: 8      # Hard limit prevents $2000 surprise bills
    spotInstanceTypes: ["t3.medium", "t3.large", "m5.large"]
    spotAllocationStrategy: "diversified"
    
  critical:
    instanceTypes: ["t3.large"]   # On-demand for critical services
    minSize: 1
    maxSize: 3

Resource Request Guidelines

resources:
  requests:
    memory: "128Mi"    # Actual usage, not theoretical
    cpu: "50m"         # 5% of CPU core
  limits:
    memory: "256Mi"    # 2x requests (good starting point)
    cpu: "200m"        # 4x requests (allows bursts)

Comparison Matrix: ArgoCD vs Flux

Criteria	ArgoCD	Flux
Memory Usage	2-4GB RAM	500MB-1GB RAM
CPU Usage	Spikes to 100% on 2 cores	Steady 10-20% on 1 core
UI Experience	Slow but functional (3-5s loads)	CLI only
Installation	1 Helm command (80% success)	Bootstrap script (mystery failures)
Debugging	UI lies, logs useless	No UI for debugging
Resource Cost	$100+/month dedicated nodes	$30-50/month shared nodes
Learning Curve	2 weeks to dangerous	1 month to competent
Production Stability	Randomly forgets applications	Rock solid until it breaks

Failed Patterns (Don't Use These)

GitOps Hooks and Sync Waves

Theory: Control deployment ordering with ArgoCD sync waves
Reality: Breaks constantly in production, more debugging than actual fixes
Alternative: Simple dependency management in Helm charts

Multi-Tenancy Through ArgoCD Projects

Theory: Isolate teams using ArgoCD projects
Reality: RBAC confusion, quota issues, debugging nightmares
Alternative: Separate clusters worth the extra cost

Automated Rollbacks Based on Metrics

Theory: Auto-rollback when SLIs drop
Reality: Requires perfect observability, never works reliably
Alternative: Manual rollbacks triggered by alerts

Essential Debugging Commands

ArgoCD Issues

# Check controller status
kubectl logs -n argocd -l app.kubernetes.io/name=argocd-application-controller --tail=100

# Test Git connectivity
kubectl exec -n argocd deployment/argocd-application-controller -- git ls-remote https://github.com/your-org/repo.git

# Check application sync status
kubectl get applications -n argocd

Pulumi Issues

# Check operator logs
kubectl logs -n pulumi-system -l app.kubernetes.io/name=pulumi-kubernetes-operator

# Check stack status
kubectl get stacks --all-namespaces
kubectl describe stack <name> -n <namespace>

General Kubernetes Debugging

# Resource utilization
kubectl top nodes
kubectl top pods --all-namespaces

# Network connectivity test
kubectl run debug --image=busybox --rm -it --restart=Never -- sh
# Inside pod: nslookup kubernetes.default.svc.cluster.local

# DNS verification
kubectl exec -ti -n kube-system <coredns-pod> -- nslookup google.com

Migration Reality Check

Timeline Expectations

• Setup Phase: 3-6 months development
• Production Readiness: Additional 3 months stabilization
• Expected Outages: 2-3 during transition
• Parallel Infrastructure Cost: $50K+ for dual environments

Prerequisites for Success

• Budget: $300+/month minimum for testing
• Team Kubernetes competency
• Comfort with GitOps philosophy requirements
• Need for deployment consistency and audit trails

When NOT to Use This Stack

• < 10 applications total
• Budget constraints (< $200/month infrastructure)
• Team new to Kubernetes
• Requirements for 100% uptime (this stack will have outages)
• Simple deployment needs

Bottom Line Assessment

Production Experience: 18 months across 3 environments
Monthly Cost: $1,200-1,500 production deployment
Incident Frequency: 2-3/month (down from 8-10 manual deployments)
Team Productivity: Dramatically improved
Setup Complexity: High (6-month migration timeline)
Operational Overhead: 2-4 hours/week platform maintenance

Recommendation: Works for complex deployments needing GitOps benefits, but requires significant investment in time, money, and expertise. Not a simple migration - plan accordingly.

Key Resource Links

Official Documentation

• Pulumi Documentation
• ArgoCD Documentation
• Kubernetes Documentation
• Helm Documentation

Critical Integration Guides

• ArgoCD with Pulumi Integration
• ArgoCD Best Practices
• EKS Best Practices

Security and Monitoring

• External Secrets Operator
• Prometheus Operator
• ArgoCD Metrics