GitOps Integration Hell: Docker + Kubernetes + ArgoCD + Prometheus

Setting up GitOps integration isn't as clean as the architecture diagrams suggest. Here's what you'll actually encounter when wiring together Docker, Kubernetes, ArgoCD, and Prometheus.

Docker: The Easy Part That Gets Complicated

Docker containers work great until you start optimizing them. Multi-stage builds will save you disk space but cost you sanity when debugging why your production image is missing dependencies that were available during the build stage.

The gotcha: Alpine-based images look appealing because they're tiny, but you'll spend hours tracking down missing `glibc` dependencies. Ubuntu images are 10x larger but actually fucking work. Choose your pain.

Version warning: Docker 20.10.17 has a known issue with BuildKit that causes random build failures on ARM64. In 2025, use Docker 27.x+ with containerd integration - it's finally stable and fixes most ARM64/Apple Silicon issues.

Kubernetes: Where Dreams Go to Die

Kubernetes networking is about as intuitive as assembly language. You'll spend your first week figuring out why pods can't reach each other despite being in the same namespace.

The learning curve: Expect 2-3 months before you stop googling "how to debug CrashLoopBackOff". Kubernetes 1.30+ improved error messages slightly, but the logs are still scattered across multiple components, and `kubectl describe` output reads like machine-generated poetry.

Memory management: If you don't set resource limits, one runaway pod will consume all available memory and crash your entire node. If you set limits too low, your perfectly working application will get OOMKilled during peak traffic.

ArgoCD: Beautiful UI, Terrible Debugging

ArgoCD's interface looks professional but becomes useless the moment something breaks. The sync status will show "Healthy" while your application returns 500 errors because ArgoCD only checks if Kubernetes accepted the manifests, not if they actually work.

Sync failures: ArgoCD will randomly stop syncing and you'll spend 3 hours troubleshooting only to discover it was a Redis connection timeout. The solution is always "restart the ArgoCD pods" but the root cause is never documented.

RBAC nightmares: Getting permissions right requires a PhD in Kubernetes RBAC. Too restrictive and deployments fail silently. Too permissive and you've violated every security policy your company has.

Prometheus: Memory Eating Monster

Prometheus collects every metric from every pod and will happily consume all your RAM doing it. The default retention policy keeps data forever, which sounds great until you realize it's storing terabytes of mostly useless metrics.

Storage nightmare: Plan for 200GB+ storage minimum. Without proper configuration, Prometheus will fill your disk in a few weeks. Set retention to 15 days unless you have infinite money for storage.

Grafana dependency: Prometheus metrics are useless without Grafana dashboards. You'll spend days creating custom PromQL queries only to discover someone already built the exact dashboard you need and published it on grafana.com.

The Repository Structure That Actually Works

Forget the textbook patterns. Here's what works in practice:

Separate app and config repos: Your application code goes in one repo, Kubernetes manifests in another. Why? Because developers will push broken YAML if you mix them, and debugging becomes impossible.

Branch-based environments are garbage: Using git branches for dev/staging/prod sounds clever but creates merge conflicts from hell. Use directory-based environments instead:

/environments/
  /prod/
  /staging/
  /dev/

Config templates save sanity: Use Helm or Kustomize. Raw YAML files mean copying the same configuration 20 times and forgetting to update half of them when values change.

Security: The Thing Everyone Ignores

Kubernetes security is an afterthought until your cluster gets compromised. Here's the minimum you need:

Image scanning: Trivy finds vulnerabilities but produces so many false positives you'll ignore them all. Configure it to only alert on "High" and "Critical" CVEs or prepare to hate your life.

RBAC hell: Every service account needs exactly the permissions it requires and nothing more. Good luck figuring out what that is without breaking everything first.

Secret management: Don't put secrets in Git even if they're "encrypted". Use External Secrets Operator or similar tools. Your future self will thank you when you're not rotating API keys after accidentally committing them to GitHub.

This stack works once you accept that it will break in creative ways and build your debugging skills accordingly.

Here's how to wire everything together without losing your mind. This took me 6 months and two production outages to figure out.

Step 1: Get Docker Images Working First

Before you even think about GitOps, make sure your Docker builds aren't garbage.

The Alpine trap: Everyone uses Alpine because it's small, then spends weeks debugging missing dependencies. Here's what actually works:

## This works but is huge
FROM node:18-bullseye-slim
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
CMD ["npm", "start"]

Multi-stage builds sound great until you realize debugging becomes impossible because your build environment is different from runtime. Use them for production, but keep a simple single-stage build for development.

Registry gotcha: Docker Hub rate limits will randomly break your builds. Use a paid registry or prepare for mysterious CI failures.

Step 2: Set Up Kubernetes (The Pain Begins)

Memory limits are not optional: Every container must have memory limits or one runaway process will kill your entire node. Start with:

resources:
  requests:
    memory: "64Mi"
    cpu: "50m"
  limits:
    memory: "128Mi"
    cpu: "100m"

Ingress is a shitshow: NGINX Ingress Controller works but the configuration syntax is insane. Expect to spend 2 days getting basic path routing working.

Namespaces matter: Don't put everything in `default`. Create separate namespaces for apps, monitoring, and ingress or you'll have permission nightmares later.

Step 3: Install ArgoCD (Where Things Get Interesting)

ArgoCD installation is deceptively simple until you need it to actually work in production:

## Don't use this in prod
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

The real config you need:

## This actually works for production
server:
  extraArgs:
    - --insecure  # Use ingress for TLS termination
  config:
    url: https://argocd.yourdomain.com
    application.instanceLabelKey: argocd.argoproj.io/instance

RBAC will make you cry: ArgoCD needs service account permissions to deploy to Kubernetes. Too much access violates security policies, too little breaks deployments. Start with cluster-admin and restrict later.

Redis dependency: ArgoCD uses Redis and doesn't tell you when it breaks. When deployments randomly stop syncing, restart the Redis pod first.

Step 4: Repository Structure That Won't Drive You Insane

Separate repos for everything:

• app-source - Your application code
• app-manifests - Kubernetes YAML files
• infrastructure - Monitoring, ingress, etc.

Why separate? Because developers will break YAML if you mix it with code. Trust me on this.

Directory structure that works:

app-manifests/
├── apps/
│   ├── frontend/
│   └── backend/
├── environments/
│   ├── dev/
│   ├── staging/
│   └── prod/
└── base/
    └── common-configs/

Use Helm or Kustomize: Raw YAML means copying configurations everywhere and forgetting to update them. Helm is more complex but handles dependencies better.

Step 5: Set Up Prometheus (Prepare for Memory Pain)

Installation using kube-prometheus-stack is straightforward but the defaults will kill your cluster:

helm install prometheus prometheus-community/kube-prometheus-stack \
  --set prometheus.prometheusSpec.retention=15d \
  --set prometheus.prometheusSpec.resources.requests.memory=8Gi \
  --set prometheus.prometheusSpec.resources.limits.memory=16Gi

Storage requirements: Prometheus with default settings will collect every metric from every pod and store it forever. Plan for 100GB+ storage minimum or your disk will fill up in weeks.

Memory consumption: Prometheus uses roughly 1-3GB RAM per million time series. A small cluster generates 500k+ series easily. Don't cheap out on memory.

Grafana dashboards: The included dashboards are decent but you'll spend days customizing them. Import dashboard 6417 for ArgoCD monitoring and save yourself hours.

Step 6: Wire Everything Together (The Integration Hell)

ArgoCD won't automatically monitor itself. You need to create ServiceMonitor resources:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argocd-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-metrics
  endpoints:
  - port: metrics

Common failure points:

• ArgoCD can't connect to Git (SSH keys expire)
• Kubernetes API rate limiting (reduce sync frequency)
• Prometheus scrape targets failing (check ServiceMonitor labels)
• Grafana datasource misconfiguration (URL must include port)

What Will Break (And How to Fix It)

ArgoCD sync failures: When ArgoCD shows "OutOfSync" but won't sync:

1. Check Git repository connectivity
2. Verify RBAC permissions on target namespace
3. Look for resource conflicts (duplicate names)
4. Restart ArgoCD server pod (works 80% of the time)

Prometheus OOM kills: When Prometheus keeps crashing:

1. Reduce retention period to 7 days
2. Increase memory limits to 16GB+
3. Use recording rules to pre-aggregate metrics
4. Consider Prometheus federation for large clusters

Networking failures: When pods can't communicate:

1. Check network policies (they block everything by default)
2. Verify ingress controller is running
3. Test DNS resolution with nslookup kubernetes.default
4. Check if service ports match container ports

This setup will work once you accept it requires constant maintenance and debugging skills.

Once you get GitOps working for one application, you'll think "this is great, let's roll it out everywhere!" That's when the real pain begins.

Multi-Cluster Reality Check

Managing multiple clusters sounds simple until you realize each cluster has its own:

• ArgoCD installation that breaks differently
• Certificate authorities that expire on different schedules
• Network policies that conflict in mysterious ways
• Kubernetes versions that drift apart over time

The authentication nightmare: Each cluster needs separate credentials. SSO integration breaks during upgrades. You'll end up with a spreadsheet of cluster URLs, admin passwords, and "which one was production again?"

Network connectivity failures: Clusters can't talk to each other half the time. VPN connections drop. Firewall rules change. DNS resolution stops working between regions.

Cost explosion: Running ArgoCD + Prometheus on every cluster multiplies your infrastructure costs. Budget $200-500/month per cluster minimum.

The Enterprise Buzzword Reality

"Platform Engineering" means your team becomes a support desk for 50 development teams who all want different things and blame you when their deployments fail.

"Self-service" means developers will somehow find ways to break things you didn't think were possible to break, then demand you fix it immediately.

"Compliance" means you'll spend 60% of your time generating reports about deployment frequencies instead of actually improving the system.

When Monitoring Becomes the Problem

Prometheus federation is supposed to solve multi-cluster monitoring. In reality, it creates a dependency nightmare where if one cluster's Prometheus dies, your entire monitoring stack becomes unreliable.

Alert fatigue is real: You'll get 500 alerts per day about pods restarting, disks filling up, and certificates expiring. After month two, you'll ignore them all, including the one that matters.

Grafana dashboard hell: Everyone wants custom dashboards. You'll have 200 dashboards that nobody uses and 5 that actually matter but are impossible to find.

The cost problem: Monitoring costs more than the applications you're monitoring. Prometheus storage, Grafana Cloud subscriptions, and log aggregation will consume 30-40% of your infrastructure budget.

Security Theater and Actual Security

Policy as code sounds great until you realize writing OPA policies requires learning a new programming language (Rego) that makes LISP look intuitive.

Image scanning produces thousands of vulnerabilities, 90% of which are false positives or in dependencies you can't update without breaking everything.

Runtime security tools like Falco generate so many alerts that you'll either ignore them all or spend full-time investigating filesystem access patterns.

The actual security risks:

• Developers will bypass all security policies when under deadline pressure
• Service accounts with cluster-admin permissions because "it's easier"
• Secrets stored in Git repositories despite all your warnings
• SSH keys that never get rotated
• Open ingress controllers with no authentication

Resource Management Nightmares

Cluster autoscaling works until traffic spikes hit, then nodes take 5 minutes to provision while your application returns 503 errors.

Resource requests and limits are either too low (causing OOMKills) or too high (wasting money). There's no middle ground.

Storage classes will bite you when you need to migrate data and discover your PVCs are locked to specific zones.

The Disaster Recovery Joke

Backup strategies work great until you try to restore from them and discover:

• Velero backups are corrupted
• Cross-region replication was never actually configured
• Your disaster recovery cluster is three Kubernetes versions behind
• The backup storage account credentials expired 6 months ago

Testing disaster recovery means breaking production to verify your backups work, which management will never approve.

The 2025 Reality Check

Platform engineering is now a recognized discipline, but that doesn't make it easier. The tools are more mature - ArgoCD ApplicationSets, Kubernetes Gateway API, Prometheus native histograms - but the fundamental complexity hasn't decreased.

What's actually improved:

• Docker ARM64 support is finally stable
• Kubernetes 1.30+ has better resource management
• ArgoCD 2.12+ is less prone to Redis failures
• Cloud provider managed services reduce operational burden

What's still broken:

• YAML configuration hell (now with even more CRDs)
• Resource limit guessing games
• Multi-cluster authentication nightmares
• Alert fatigue from monitoring everything

This is the reality of running GitOps at scale. It works, but requires a team of full-time engineers and a realistic understanding that everything will break in ways you never anticipated. Budget accordingly, hire people who enjoy debugging at 3am, and accept that your first production outage is a learning opportunity, not a failure.