ArgoCD Production Troubleshooting: AI-Optimized Knowledge Base

Critical Failure Scenarios and Solutions

Memory-Related Failures

Repository Server Out-of-Memory (OOMKilled)

• Trigger: Default 500MB limit with large Helm charts or monorepos
• Impact: Sync failures across all applications, complete deployment halt
• Immediate Fix: Increase memory limit to 2-4GB
• Root Cause: Caching rendered manifests for hundreds of applications simultaneously
• Breaking Point: 8GB+ RAM consumption observed with monorepos containing 300+ microservices
• Long-term Solution: Split monorepos into smaller repositories (<20 services each)

spec:
  template:
    spec:
      containers:
      - name: repo-server
        resources:
          limits:
            memory: 2Gi  # Up from default 500Mi

Redis Memory Explosion (4GB+ consumption)

• Trigger: Hundreds of applications with large Git repos
• Detection: kubectl exec -it redis-pod -- redis-cli info memory
• Emergency Fix: Restart Redis pod (cache loss acceptable)
• Permanent Solution: Enable compression and reduce cache TTL

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm
data:
  redis.compress: "gzip"
  repo.server.ttl: "5m"

Sync Operation Failures

Applications Stuck Syncing (No Error Messages)

• Root Cause: GitHub API rate limits (90% frequency)
• Math: 200 apps × 20 polls/hour = 96,000 requests/day vs 60 requests/minute limit
• Detection: Check application-controller logs for "API rate limit exceeded"
• Immediate Fix: Increase polling interval to 10+ minutes
• Scalable Solution: Implement webhook-based sync + multiple GitHub tokens

Auto-Sync Ignores Changes (Manual Sync Works)

• Root Cause: Resource perpetually out-of-sync causing sync loop failure
• Detection: argocd app diff app-name shows persistent differences
• Common Culprits: ConfigMaps with generated data, operator-managed resources, finalizer issues
• Solution: Add ignore annotations for operator-managed fields

metadata:
  annotations:
    argocd.argoproj.io/compare-options: IgnoreExtraneous

Resource Hook Failures

Hooks Timeout/Never Complete

• Default Timeout: 5 minutes (too short for database migrations)
• Impact: Applications stuck in "Progressing" status indefinitely
• Solution: Increase timeout + proper cleanup policy

apiVersion: batch/v1
kind: Job
metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  activeDeadlineSeconds: 1200  # 20 minutes
  backoffLimit: 0  # Don't retry failed migrations

Performance Thresholds and Breaking Points

Scaling Limits

• Single ArgoCD Instance: ~100 applications (comfortable), 500+ applications (UI becomes sluggish)
• Memory Requirements: 2-4MB RAM per managed application (repository server)
• Git Polling: Scales linearly with application count, hits API limits at 200+ apps
• Multi-Cluster: Each cluster adds ~50MB baseline memory usage

UI Performance Degradation

• Breaking Point: >1000 applications renders UI effectively unusable
• Load Time: >10 seconds for application list with 200+ apps
• Solution: Enable application list paging, use CLI for bulk operations

Production Incident Patterns

The Great Prune Disaster

• Scenario: Auto-sync with prune enabled globally
• Impact: Deleted external secrets, service mesh sidecars, monitoring agents, persistent volumes
• Duration: 4 hours downtime, customer data loss
• Prevention: Never enable prune globally, use strict RBAC policies

spec:
  syncPolicy:
    automated:
      prune: false    # NEVER enable globally
      selfHeal: true  # Auto-fix drift only

Memory Leak Weekend Incident

• Trigger: Monorepo with 300+ microservices, complex Helm charts
• Escalation: 12GB memory consumption → OOMKilled → sync failures across all apps
• Resolution: Split monorepo into 15 smaller repos (~20 services each)
• Lesson: Monorepo size directly correlates with memory consumption

Multi-Cluster Authentication Failures

• Frequency: Random clusters become "Connection Refused" after 6+ months
• Causes: EKS/GKE API endpoints change, service account tokens expire, network policies update
• Detection: Automated health checks every 5 minutes
• Auto-Remediation: Script to re-register clusters with fresh tokens

Resource Requirements and Sizing

Production-Ready Configuration

# Repository Server Scaling
spec:
  replicas: 5  # Up from default 1
  template:
    spec:
      containers:
      - name: repo-server
        resources:
          limits:
            memory: 2Gi    # Up from 256Mi
            cpu: 1000m     # Up from 250m
          requests:
            memory: 1Gi
            cpu: 500m

Redis Optimization for Scale

# Redis with persistence and memory optimization
containers:
- name: redis
  resources:
    limits:
      memory: 4Gi     # Large cache for 500+ apps
    requests:
      memory: 2Gi
  args:
  - redis-server
  - --maxmemory
  - 3gb
  - --maxmemory-policy
  - allkeys-lru       # Evict least recently used keys

Critical Monitoring Metrics

Application Health Indicators

• argocd_app_health_status{health_status!="Healthy"}: Alert when apps unhealthy >10 minutes
• argocd_app_sync_total{phase="Failed"}: Track sync failure rate trends
• container_memory_working_set_bytes{pod=~"argocd-.*"}: Memory consumption alerts at 80% limit

Performance Baselines

• Average Sync Duration: Track histogram_quantile(0.95, rate(argocd_app_sync_bucket[5m]))
• Git Operation Success Rate: Monitor fetch failures by repository
• API Response Times: Track degradation under load

Implementation Decision Criteria

When to Split ArgoCD Instances

• >500 applications: UI performance degrades significantly
• Critical vs non-critical apps: Isolate production workloads
• Multi-team environments: Separate RBAC boundaries

Repository Structure Decisions

• Monorepo threshold: >50 services in single repo causes memory issues
• Polling vs Webhooks: Webhooks required above 100 applications to avoid rate limits
• Helm complexity: Charts with >50 templates require increased memory limits

Failure Mode Prevention

Default Settings That Fail in Production

• Memory limits: 500MB default insufficient for real workloads
• Sync timeout: 5 minutes too short for complex deployments
• Polling frequency: 3 minutes causes API rate limit issues
• Auto-prune: Extremely dangerous when enabled globally

Critical Warnings

• ArgoCD "Healthy" ≠ Application Working: Health checks only verify Kubernetes resources exist
• Webhook dependencies: GitHub/GitLab webhooks fail silently, require monitoring
• Multi-cluster token expiry: Service account tokens expire unexpectedly
• Redis restart impacts: Cache loss acceptable but causes temporary performance degradation

Emergency Response Procedures

Immediate Triage Commands

# Check ArgoCD component health
kubectl get pods -n argocd
kubectl top pods -n argocd

# Identify stuck applications
argocd app list --output name | xargs -I {} argocd app get {}

# Memory usage investigation
kubectl exec -it redis-pod -- redis-cli info memory
kubectl logs -n argocd deployment/argocd-repo-server --tail=100

Recovery Procedures

1. Memory exhaustion: Restart affected component, increase limits
2. Sync failures: Check API rate limits, increase polling intervals
3. Authentication issues: Re-register clusters with fresh tokens
4. Hook timeouts: Increase activeDeadlineSeconds, implement proper cleanup

Resource Quality Assessment

Tool Reliability Indicators

• High-quality documentation: Official ArgoCD troubleshooting guide comprehensive
• Active community: 15,000+ Slack members, GitHub issues actively maintained
• Breaking changes: Major version upgrades require migration planning
• Production readiness: Stable after v2.0, but requires careful configuration tuning

Support Channels Quality

• ArgoCD Slack #argo-cd: Real-time community support, high response rate
• GitHub Issues: 3,000+ resolved issues, search before creating new
• Stack Overflow: Technical Q&A with detailed solutions
• Official Documentation: Comprehensive but assumes Kubernetes expertise

Migration Considerations

• Version upgrades: Test in staging first, breaking changes in major versions
• Multi-cluster complexity: Exponentially increases operational overhead
• Operator integration: Requires careful ignore configurations for managed resources
• Security hardening: Default installation not production-ready, requires additional configuration