GitOps Integration: Docker, Kubernetes, ArgoCD, Prometheus - AI Technical Reference

Configuration

Docker Container Setup

Production-Ready Image Configuration:

# Reliable base - Ubuntu over Alpine
FROM node:18-bullseye-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
CMD ["npm", "start"]

Critical Settings:

• Use Docker 27.x+ with containerd integration for ARM64 stability
• Avoid Alpine for high-level languages (Python, Node) - missing glibc dependencies cause failures
• Multi-stage builds optimize size but break debugging - use single-stage for development
• Docker Hub rate limits break builds - use paid registry or expect mysterious CI failures

Kubernetes Resource Management

Essential Memory/CPU Limits (Required - Not Optional):

resources:
  requests:
    memory: "64Mi"
    cpu: "50m"
  limits:
    memory: "128Mi"
    cpu: "100m"

Namespace Strategy:

• Separate namespaces for apps, monitoring, ingress
• Never use default namespace - creates permission conflicts
• Network policies block everything by default

Storage Requirements:

• Plan 100GB+ minimum for Prometheus metrics
• Use gp3 storage class on AWS (cheaper than gp2)
• PVCs lock to specific zones - impacts disaster recovery

ArgoCD Production Configuration

Working Production Setup:

server:
  extraArgs:
    - --insecure  # Use ingress for TLS termination
  config:
    url: https://argocd.yourdomain.com
    application.instanceLabelKey: argocd.argoproj.io/instance

redis:
  resources:
    requests:
      memory: 256Mi
    limits:
      memory: 512Mi
  persistence:
    enabled: true
    size: 1Gi

RBAC Requirements:

• Start with cluster-admin, restrict later
• Service accounts lose permissions during cluster upgrades
• ArgoCD needs specific ServiceMonitor resources for Prometheus integration

Prometheus Resource Configuration

Memory/Storage Settings (Critical):

prometheus:
  prometheusSpec:
    resources:
      requests:
        memory: 8Gi
      limits:
        memory: 16Gi
    retention: 15d  # Default forever retention fills disk
    scrapeInterval: 60s  # Default 15s is overkill

global:
  scrape_interval: 60s

Storage Planning:

• 1-3GB RAM per million time series
• Small cluster generates 500k+ series
• 200GB+ storage minimum with 15-day retention
• Default retention (forever) fills disk in weeks

Resource Requirements

Time Investment

• Basic setup (experienced): 1-2 weeks
• First-time implementation: 2-3 months
• Production-ready with monitoring: 3-6 months
• Debugging proficiency: 6-12 months

Cost Structure (2025 AWS Pricing)

• EKS cluster: $73/month
• 3 t3.medium nodes: $105/month
• Application Load Balancer: $27/month
• EBS storage (100GB gp3): $8/month
• NAT gateway: $45/month each
• Total minimum: $215-320/month per cluster
• Multi-cluster multiplication: $200-500/month per additional cluster

Human Resources

• Requires team of full-time engineers for production scale
• 24/7 on-call support for debugging failures
• Platform engineering becomes support desk for development teams

Critical Warnings

ArgoCD Failure Patterns

Primary Issue: Redis dependency failures cause silent sync停止

• ArgoCD shows "Healthy" while applications fail
• Sync status checks Kubernetes acceptance, not application functionality
• Fix: kubectl rollout restart deployment argocd-server -n argocd

Authentication Breaks: During cluster upgrades

• Admin passwords: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Prometheus Memory Consumption

Default Configuration Kills Clusters:

• Collects every metric from every pod forever
• No memory limits = OOMKill entire nodes
• Uses 1-3GB RAM per million time series
• Solution: Set 15-day retention, 8-16GB memory limits

Docker Image Issues

Alpine Linux Problems:

• Missing glibc dependencies for most applications
• Debugging becomes impossible with different build/runtime environments
• Ubuntu images 10x larger but actually work

Kubernetes Networking Failures

Common Failure Points:

• Pods can't communicate despite same namespace
• Ingress returns 503 - check service port mapping
• Network policies block everything by default
• DNS resolution fails between clusters

Security Theater vs Reality

Policy Enforcement Failures:

• Developers bypass security under deadline pressure
• Image scanning: 90% false positives
• Service accounts get cluster-admin "for convenience"
• SSH keys never rotated
• Secrets stored in Git repositories

Breaking Points and Failure Modes

Scale-Related Failures

Multi-Cluster Management:

• Each cluster breaks differently
• Certificate expiration on different schedules
• Network connectivity failures between regions
• Authentication nightmare with separate credentials per cluster

Monitoring Overload:

• 500+ alerts per day lead to alert fatigue
• Monitoring costs 30-40% of infrastructure budget
• Prometheus federation creates dependency chains
• Grafana dashboard proliferation (200 unused, 5 critical)

Resource Exhaustion Patterns

Memory Issues:

• One misconfigured pod consumes all cluster memory
• Prometheus without limits fills all disk space
• Too many metrics collectors overload Kubernetes API

Storage Problems:

• Backup storage credentials expire unnoticed
• Cross-region replication misconfigured
• Velero backups corrupted during restore attempts
• Disaster recovery clusters lag behind production versions

Version-Specific Issues

Docker 20.10.17: BuildKit random failures on ARM64
Kubernetes 1.30+: Gateway API replaces Ingress (migration required)
ArgoCD 2.8.x: Redis dependency issues (improved in 2.12+)
Prometheus 2.50+: Native histograms require storage migration

Emergency Procedures

ArgoCD Rollback Process

When ArgoCD Works:

git revert <commit-hash>
git push
argocd app sync myapp --force

When ArgoCD Broken:

kubectl apply -f previous-working-config.yaml
kubectl rollout undo deployment/myapp

Nuclear Option:

kubectl delete deployment myapp
kubectl apply -f last-known-good-config.yaml

Prometheus Recovery

OOMKill Recovery:

1. Reduce retention to 7 days
2. Increase memory limits to 16GB+
3. Implement recording rules for metric aggregation
4. Consider Prometheus federation for large clusters

Common Debugging Commands

Pod Failures:

kubectl logs <pod> --previous
kubectl describe pod <pod>
kubectl auth can-i create deployments --as=system:serviceaccount:argocd:argocd-application-controller

Service Connectivity:

kubectl get endpoints <service-name>
kubectl exec -it <pod> -- curl http://service-name:port
nslookup kubernetes.default

Repository Structure Patterns

Working Directory Layout

app-manifests/
├── apps/
│   ├── frontend/
│   └── backend/
├── environments/
│   ├── dev/
│   ├── staging/
│   └── prod/
└── base/
    └── common-configs/

Anti-Patterns:

• Branch-based environments (merge conflict hell)
• Mixed application code and Kubernetes manifests
• Raw YAML without templating (Helm/Kustomize)

Secret Management

External Secrets Operator Configuration:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: app-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: app-secrets
  data:
  - secretKey: db-password
    remoteRef:
      key: myapp/db
      property: password

Integration Issues:

• Requires vault/AWS permissions setup
• Secrets don't auto-refresh on source changes
• ArgoCD won't display secret contents (security feature)

Technology Stack Comparison

Component	ArgoCD + Prometheus	Flux + Grafana	Jenkins X + Tekton	GitLab + Kubernetes
Maturity	CNCF Graduated	CNCF Incubating	Medium (v3 rebuild)	High Enterprise
Learning Curve	Moderate UI	Steep CRDs	Complex Components	Integrated Platform
Multi-cluster	Native Support	Multi-tenancy	Multiple Contexts	Environment-based
CI Integration	External Required	External + Image Automation	Built-in Tekton	Integrated GitLab CI

Decision Criteria:

• ArgoCD: Best for teams wanting GitOps-first approach with familiar UI
• Flux: Best for Kubernetes-native teams comfortable with CRDs
• Jenkins X: Best for teams needing integrated CI/CD pipeline
• GitLab: Best for teams already using GitLab ecosystem

This technical reference provides the operational intelligence needed for successful GitOps implementation while avoiding the common failure modes that cause production outages.