Jenkins + Docker + Kubernetes: How to Deploy Without Breaking Production (Usually)

Here's the thing nobody tells you: Jenkins is a fucking dinosaur from 2005 that somehow became the backbone of half the internet's deployments. Docker is simple until you need to debug networking. And Kubernetes is powerful but will consume your entire DevOps team's time.

But they work together, and if you do it right, you can deploy code without breaking production. Usually.

The Real Architecture (Not the Marketing Bullshit)

Jenkins is your build orchestrator - it's like the anxious project manager that keeps checking if everything's done. Docker packages your app into containers so it runs the same everywhere (in theory). Kubernetes is the cluster manager that's supposed to keep everything running but has opinions about literally everything.

Here's what actually happens: Developer pushes code → Jenkins freaks out and starts a build → Docker builds an image (hopefully) → Jenkins runs tests (which fail for mysterious reasons) → If everything passes, Kubernetes gets the image and tries to deploy it → Something breaks → You debug for 3 hours → Repeat.

I spent 6 months setting this up at my last job. The official docs are basically useless for the actual problems you'll hit.

Current State (September 2025): What's Actually Changed

The ecosystem keeps evolving, and not always for the better. Jenkins 2025 versions still have security issues coming out every few months. Kubernetes 1.31 is the current stable release, but if you're on cloud providers, you're probably stuck on whatever version they decide to support.

Docker's still Docker - works great until it doesn't. The main difference now is that everyone's trying to replace it with Podman or containerd, which just adds another layer of complexity to debug.

Jenkins: Maximum Flexibility, Maximum Pain

Jenkins has plugins for everything. That's both its strength and its curse. You'll start with a simple pipeline and end up with 47 plugins that all need different versions and break when you update anything.

The Kubernetes plugin sounds great - dynamic agents that spin up as pods! What they don't mention is that these agents randomly fail to connect, eat CPU like crazy, and the logs are completely useless when debugging.

Pro tip: Use pipeline-as-code (Jenkinsfiles) or you'll lose your sanity maintaining freestyle jobs. Learned this the hard way when we had 200+ jobs and no idea what any of them actually did.

Docker: Simple Until It's Not

Docker containers are supposed to solve "works on my machine" problems. They do, mostly. But then you hit networking issues, and suddenly you're reading RFC documents at 2am trying to understand bridge networks.

Docker runs a daemon in the background that handles all the container stuff. When it crashes (and it will), everything stops working until you restart it.

Docker builds work great until your disk fills up with layers. Set up layer caching or your builds will take forever. Also, multistage builds are mandatory - nobody wants 2GB images in production.

The Docker daemon loves to randomly stop working. The universal fix is restart, which works about 80% of the time. The other 20%, you'll be googling cryptic error messages.

Kubernetes: The Overengineered Beast

Kubernetes can do everything. That's the problem - it's like using a nuclear reactor to heat your coffee. Most teams need maybe 10% of its features but spend 90% of their time fighting YAML files.

Kubernetes has a bunch of control plane services that coordinate everything. When any of them breaks, you'll get vague error messages that help nobody.

RBAC is like playing permission bingo. Everything fails with vague "forbidden" errors until you add the magic annotation that makes it work. The cluster will be fine for weeks, then suddenly nothing can pull images and you'll spend a day figuring out imagePullSecrets.

Pod startup times are unpredictable. Sometimes pods start in 10 seconds, sometimes 5 minutes. The scheduler has opinions you didn't know existed.

What Actually Works in Production

After breaking production more times than I care to count, here's what actually works:

1. Keep Jenkins simple - Don't install every plugin. Each one is a potential failure point.
2. Docker layer caching saves your sanity - Builds that take 2 minutes vs 20 minutes matter at scale.
3. Kubernetes resource limits are mandatory - One pod eating all the CPU will take down your entire node.
4. Rolling deployments with readiness probes - Kubernetes won't send traffic to broken pods, usually.
5. Separate CI and CD - Jenkins builds and tests, something else (like ArgoCD) handles deployment.

The dirty secret: Most successful teams use Jenkins for CI and something simpler for CD. Kubernetes is great for running apps, terrible for deployment automation.

The Reality Check: What Success Actually Looks Like

Here's what a working setup looks like after 2 years of iteration:

Jenkins runs lightweight - No builds on the master, agents spin up for specific tasks and die. Pipeline libraries contain all the common patterns so teams don't write the same Groovy bullshit 50 times.

Docker images are boring - Alpine-based, multi-stage builds, and under 200MB. The fancy optimizations matter less than consistency.

Kubernetes clusters are cattle, not pets - Immutable infrastructure with everything in Git. When shit breaks, you replace it, not fix it.

The teams that succeed treat this stack like plumbing - boring, reliable, and invisible. The ones that fail get distracted by the latest Kubernetes features instead of focusing on shipping code.

Look, theory is great, but when your deployment is down at 3am and the CEO is asking questions, you need solutions that actually work. Here's what breaks, why it breaks, and what actually fixes it.

After debugging this shit for 3+ years in production, I can tell you the patterns. 80% of outages come from the same 5 problems. The other 20% are creative new ways for things to fail.

Jenkins Agents: The Biggest Source of Pain

Jenkins agents in Kubernetes pods sound great - they scale automatically! In reality, they randomly fail to connect and debugging them is hell because the logs tell you nothing useful.

Jenkins agents are supposed to be simple. They're not. I've debugged more agent connection issues than I care to count.

Agent won't connect? Check these in order:

1. `kubectl get pods -n jenkins` - Is the pod even running?
2. `kubectl describe pod ` - Look for "ImagePullBackOff" or "CrashLoopBackOff"
3. `kubectl logs ` - Usually says "connection refused" which helps nobody

The real fix: The Jenkins service account probably doesn't have the right RBAC permissions. You need `create`, `get`, `list`, `watch`, `update`, `patch`, `delete` on pods. And probably more because Kubernetes loves granular permissions.

Agent randomly dies during builds? Memory limits. Every fucking time. Kubernetes kills pods that exceed memory limits without warning. Set `resources.requests.memory` and `resources.limits.memory` in your pod template, or watch your builds fail randomly.

resources:
  requests:
    memory: \"1Gi\"
    cpu: \"500m\"
  limits:
    memory: \"2Gi\" 
    cpu: \"1000m\"

Docker Daemon Issues That Will Ruin Your Day

"Cannot connect to the Docker daemon" errors? Three possible causes:

1. Docker daemon isn't running - `sudo systemctl restart docker` fixes this 80% of the time
2. Permission issues - Add the jenkins user to the docker group, or mount the docker socket properly
3. Docker daemon crashed - Check `/var/log/docker.log` for out of memory or disk space issues

Builds randomly fail with "no space left on device"? Docker images pile up like dirty laundry. Clean them:

docker system prune -a
docker volume prune

Set this up as a cron job or your disk will fill up guaranteed.

Docker builds taking 20+ minutes? Layer caching is fucked. Either your Dockerfile is written badly (put the least changing stuff first), or your build context is huge. Add a `.dockerignore` file:

node_modules/
.git/
*.log
tmp/

Kubernetes: Where Good Builds Go to Die

Pods stuck in \"Pending\" status? Resources. The scheduler can't find a node with enough CPU/memory. Check with:

kubectl describe pod <pod-name>
## Look for events like \"Insufficient memory\" or \"Insufficient cpu\"

Fix: Either add more nodes or reduce resource requests. Or delete the pod that's eating all your CPU (you know the one).

"ImagePullBackOff" errors? Three common causes:

1. Registry authentication failed - Your imagePullSecrets are wrong or missing
2. Image doesn't exist - Typo in image tag, or build actually failed but Jenkins said it succeeded
3. Network issues - Nodes can't reach your registry (firewall/DNS problems)

Debug with: `kubectl describe pod ` and look at the events.

Deployments stuck at \"0/3 ready\"? Readiness probe is failing. Your app is starting but the health check endpoint returns 500. Check:

kubectl logs <pod-name>
kubectl exec <pod-name> -- curl localhost:8080/health

Usually the app is crashing on startup and you'll find the real error in the logs.

The Network Is Always the Problem

Services can't reach each other? Kubernetes networking is black magic that works until it doesn't. Debug steps:

1. `kubectl get pods -o wide` - Are pods actually running?
2. `kubectl get svc` - Does the service exist and have endpoints?
3. `kubectl describe svc ` - Check the selector matches your pod labels
4. `kubectl exec -- nslookup ` - DNS working?

If DNS is broken, restart CoreDNS: `kubectl rollout restart deployment/coredns -n kube-system`

Can't push to your Docker registry? Network policies or firewall rules. Test from inside the cluster:

kubectl run debug --rm -i --tty --image=alpine -- sh
## Then try: wget <registry-url>

Resource Limits: The Silent Build Killer

Set resource limits on everything or one rogue pod will bring down your entire node. I learned this when a memory leak in a build took out our entire Kubernetes cluster at 2am.

resources:
  requests:
    memory: \"512Mi\"
    cpu: \"250m\"
  limits:
    memory: \"1Gi\"
    cpu: \"500m\"

Cluster nodes constantly running out of resources? Check what's actually using them:

kubectl top nodes
kubectl top pods --all-namespaces

Usually it's old completed job pods that never got cleaned up, or someone deployed something that ignores resource limits.

The Nuclear Option: When All Else Fails

Sometimes you need to blow things up and start over:

1. Jenkins agent won't work? Delete it: `kubectl delete pod `
2. Deployment stuck? Force recreate: `kubectl rollout restart deployment/`
3. Entire namespace fucked? Nuclear option: `kubectl delete namespace ` (careful with this one)
4. Docker daemon possessed by demons? `sudo systemctl restart docker`
5. Kubernetes cluster in weird state? Reboot nodes one by one

What Actually Prevents These Problems

Monitoring that matters:

• Set up alerts for pods crashing, not just "cluster healthy"
• Monitor disk space on all nodes (Docker images fill disks fast)
• Alert when any namespace uses >80% of resource quotas
• Track build success rates and failure reasons

Resource management:

• Set resource requests/limits on EVERYTHING
• Use namespace resource quotas to prevent teams from eating all resources
• Clean up old images and completed jobs automatically

Real load testing:

• Your pipeline works fine with 5 builds, breaks at 50. Test at scale.
• Network policies that work in dev might break in prod. Test the actual network paths.

The truth nobody tells you: Most production issues are resource exhaustion or permissions. Fix those two and you'll prevent 80% of the pain.

The Lessons That Cost Me Sleep (So You Don't Have To Learn Them)

Lesson 1: Your staging environment is a lie. It works perfectly, then production breaks in creative ways. The only real test is production load with production data and production stupidity.

Lesson 2: Jenkins plugin updates will break your pipeline. Pin versions or accept that builds will randomly fail after updates. There's no middle ground.

Lesson 3: Kubernetes is eventually consistent until it's not. That deployment that's been "pending" for 20 minutes? It's not coming back without intervention.

Lesson 4: Docker layer caching is magic until your disk fills up. Then everything breaks at once and you spend a weekend fixing it.

Lesson 5: The problem is always networking. Always. Even when it's clearly not networking, it's somehow still networking.

The pattern: Simple solutions work. Complex solutions create new problems. The best architecture is the one that lets you sleep at night.