CI/CD Pipeline Troubleshooting - When Everything Goes to Hell at 3AM

When your build is broken and everyone's breathing down your neck, here's the order that actually saves time:

1. Is It "Works on My Machine" Syndrome?

The most infuriating and common cause of pipeline failures. Your code runs perfectly locally but dies in CI with errors that make no fucking sense.

Quick test: Can you reproduce the failure locally? If not, it's an environment difference, and you're in for a long night.

## Copy the exact environment from CI
docker run -it --rm \
  -e NODE_ENV=production \
  -e DATABASE_URL=postgres://user:pass@db:5432/myapp \
  node:18-alpine \
  sh

Common gotchas I've learned the hard way:

• Case sensitivity: Your MacBook doesn't care about file case, Linux does. import './Component' works locally but fails in CI when the file is component.tsx
• Path separators: Windows uses \, everything else uses /. Hardcoded paths break across platforms
• Environment variables: Your .env file works locally but CI doesn't have access to it
• Node version drift: You're running Node 18.2.0 locally, CI is using 18.1.0, and that patch version broke something

Last month I spent 4 hours debugging why our tests passed locally but failed in CI with Cannot resolve module '@/components/Button'. Turns out the CI image was missing the TypeScript path mapping configuration. The fix was adding tsconfig.json to our Docker build context.

One line in the Dockerfile. One fucking line. Four hours of my life I'll never get back.

But you know what? Finding that bug felt incredible. Like solving a puzzle that's been taunting you. That moment when the build finally goes green after hours of red failures? Pure dopamine hit. This is why we do this job.

Recent gotcha: Newer Node versions sometimes change how module resolution works. If you're getting ERR_MODULE_NOT_FOUND errors that work fine in older versions, check if your CI is pulling a different Node version than expected:

## Check exactly what Node version CI is using
node --version
npm --version

## If using nvm in CI, pin the version
echo "22.9.0" > .nvmrc
nvm use

2. Check the Obvious Shit First

Has anything changed recently? I know, I know, "nothing changed" - but check anyway:

• Package updates in the last 48 hours
• Environment variable changes
• Infrastructure updates (Node version, base Docker images)
• New team members who might have merged something

## Check recent commits that might be cursed
git log --oneline --since="2 days ago"

## See what packages changed
git diff HEAD~5 package.json
git diff HEAD~5 package-lock.json

Are the dependencies actually installing?

• npm install hanging for 30+ minutes usually means network issues or registry problems
• pip install failing with SSL errors means your base image is too old
• bundle install timing out means RubyGems is having a bad day

3. Resource Limits: The Silent Killer

Your pipeline was working fine until your codebase grew or your test suite got bigger. Now builds randomly fail with unhelpful error messages.

Memory exhaustion symptoms:

• JavaScript heap out of memory during webpack builds
• Killed with no other context (Linux OOM killer)
• Tests that pass individually but fail when run together
• Docker builds that hang during npm install

## Check if you're hitting memory limits
docker stats --no-stream

## Increase Node.js heap size for builds
export NODE_OPTIONS="--max-old-space-size=4096"

I learned this the hard way when our webpack build started failing after adding React Query to our app. The build would get to 90% compilation and just... die. Killed. No error message, no stack trace, no helpful context.

Six fucking hours. Six hours of my Saturday debugging this while my family went to the beach without me. Tried different Node versions, cleared all caches, rebuilt the Docker image from scratch. Nothing.

Finally ran docker stats and saw memory usage spike to 2GB and flatline. The CI runner was silently OOM-killing the process. Bumped memory to 4GB in our GitHub Actions config and boom - build worked perfectly.

Sometimes I hate this job. But sometimes it's kind of beautiful how simple the solution is once you find it.

4. Network Connectivity Hell

Your CI environment might not have access to the same resources as your development machine.

Common network failures:

• Corporate firewalls blocking package registries
• DNS resolution issues in containerized environments
• Registry authentication failures
• Proxy configuration problems

## Test network connectivity from your CI environment
curl -I https://registry.npmjs.org/
nslookup github.com
ping google.com

The registry authentication nightmare: Your personal access token works locally but fails in CI because you're using a different account or the token has different permissions.

5. Timing and Race Conditions

Some failures only happen under specific timing conditions that are more likely in CI environments.

Classic race conditions:

• Tests that depend on database seeds not being fully applied
• File system operations that assume synchronous completion
• Network requests without proper timeout handling
• Process startup dependencies that aren't properly awaited

// This works locally but fails in CI
const server = require('./server');
const request = require('supertest');

// Server might not be ready yet
test('should return 200', async () => {
  const response = await request(server).get('/health');
  expect(response.status).toBe(200);
});

// Better: wait for server to be ready
test('should return 200', async () => {
  await server.ready(); // Wait for server startup
  const response = await request(server).get('/health');
  expect(response.status).toBe(200);
});

The Nuclear Debugging Options

When nothing else works and you're getting desperate:

SSH into the CI runner (if possible):

## For self-hosted runners
docker exec -it runner-container bash

## For GitHub Actions with tmate
- name: Debug with tmate
  uses: mxschmitt/action-tmate@v3

Enable verbose logging everywhere:

## Docker build debug
DOCKER_BUILDKIT=0 docker build --progress=plain --no-cache .

## npm debug output
npm install --loglevel=verbose

## GitHub Actions debug
export ACTIONS_STEP_DEBUG=true
export ACTIONS_RUNNER_DEBUG=true

Reproduce the exact CI environment locally:

## Pull the exact same Docker image
docker pull node:18-alpine

## Run with the same environment variables
docker run -it --rm \
  -e CI=true \
  -e NODE_ENV=production \
  -v $(pwd):/app \
  -w /app \
  node:18-alpine \
  /bin/sh

6. New 2025 Debugging Scenarios (The Fresh Hell)

GitHub Actions with Apple Silicon Runners: GitHub now offers M1/M2 runners, but Docker builds targeting x86 fail in weird ways:

## This breaks on M1 runners with Docker buildx
- name: Build for production
  run: docker build --platform linux/amd64 .

## This works
- name: Build for production
  run: | 
    docker buildx create --use
    docker buildx build --platform linux/amd64 --load -t myapp .

Bun/Deno in CI: New JavaScript runtimes cause compatibility hell. I wasted a perfectly good weekend recently debugging why our Next.js app worked locally but failed CI with import.meta is not defined.

Turned out our CI image had both Node and Bun installed, and some fucking dependency was calling Bun instead of Node to run our build script. Two days of my life debugging a build tool I wasn't even trying to use.

Fixed it by explicitly setting engines.node in package.json and removing Bun from the CI image. But honestly? Bun is pretty sweet when it works. Might try it again when it's more stable.

pnpm Workspace Issues: pnpm is becoming popular, but workspace setups break differently than npm:

## This works locally but fails in CI
pnpm install

## CI needs workspace protocol understanding
pnpm install --frozen-lockfile --prefer-offline

Podman Instead of Docker: Some CI providers switched to Podman. Commands look the same but authentication breaks:

## Docker way
docker login -u $USER -p $PASS registry.com

## Podman way (different auth mechanism)
podman login --username $USER --password $PASS registry.com

The key to not losing your mind is being systematic. Most pipeline failures fall into these categories, and checking them in order saves hours of random debugging.

Container orchestration was supposed to make deployments easier. Instead, we traded one set of problems for a completely different set of problems that require specialized knowledge to debug. Here's how to fix the most common disasters.

Docker Build Performance Hell

Your Docker build takes 20 minutes and you're losing your mind:

The problem is usually layer caching and build context size. Docker has to send your entire project directory to the build daemon, then rebuild layers that could be cached.

## This Dockerfile will make you suffer
FROM node:18
COPY . /app
WORKDIR /app
RUN npm install
RUN npm run build

## This one won't ruin your day
FROM node:18-alpine as dependencies
WORKDIR /app
## Copy only package files first for better caching
COPY package*.json ./
RUN npm ci --only=production

FROM node:18-alpine as build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:18-alpine as runtime
WORKDIR /app
COPY --from=dependencies /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
COPY package*.json ./
EXPOSE 3000
CMD ["npm", "start"]

The .dockerignore file you forgot to create:

node_modules
npm-debug.log
.git
.DS_Store
*.md
.env
coverage/
.nyc_output

I learned this the hard way when our build went from 2 minutes to 15 minutes after someone added a logs/ directory with 2GB of log files. Docker was copying the entire thing to the build context every single time. Adding .dockerignore fixed it instantly.

Multi-Platform Build Failures

Building for ARM64 (Apple Silicon) when your CI runs AMD64 causes weird failures:

## This breaks on ARM Macs
FROM node:18

## This works everywhere
FROM --platform=$BUILDPLATFORM node:18

## Or be explicit about what you want
FROM --platform=linux/amd64 node:18

If you need multi-platform builds:

## Use buildx for multi-platform images
FROM --platform=$BUILDPLATFORM node:18 as builder
## ... build steps ...

FROM node:18-alpine
COPY --from=builder /app/dist ./dist

Kubernetes Deployment Debugging: The Systematic Approach

When your deployment fails, Kubernetes gives you cryptic errors spread across multiple resources. Here's how to actually figure out what's wrong:

Step 1: Start with the deployment status

## See if pods are even being created
kubectl get deployments
kubectl describe deployment your-app

## Check replica sets (manages pods for deployments)
kubectl get replicasets
kubectl describe replicaset your-app-xxx

Step 2: Check pod status

## See what state your pods are in
kubectl get pods -l app=your-app

## Get detailed info about failed pods
kubectl describe pod your-app-xxx

## Check events (this is where the real info usually is)
kubectl get events --sort-by=.metadata.creationTimestamp

Step 3: Dive into logs

## Current container logs
kubectl logs your-app-xxx

## Previous container logs (if it crashed and restarted)
kubectl logs your-app-xxx --previous

## Follow logs in real-time
kubectl logs -f deployment/your-app

The Most Common Kubernetes Fuckups

Resource limits that are too restrictive:

Your app needs 512MB to start but you set limits to 256MB. Kubernetes kills it immediately.

## This will kill your Node.js app
resources:
  limits:
    memory: "128Mi"  # Too small for most Node apps
    cpu: "100m"      # Too small for build processes

## This actually works
resources:
  requests:
    memory: "256Mi"
    cpu: "100m"
  limits:
    memory: "512Mi"   # Room for memory spikes
    cpu: "500m"       # Room for CPU spikes

Liveness probes that are too aggressive:

Your app takes 30 seconds to start but the probe checks every 10 seconds with no grace period.

## This kills apps that take time to start
livenessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 5    # Too soon!
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 2       # Only 2 failures = death

## This actually works
livenessProbe:
  httpGet:
    path: /health
    port: 3000
  initialDelaySeconds: 60   # Give it time to start
  periodSeconds: 30
  timeoutSeconds: 10
  failureThreshold: 3       # More forgiving

Readiness probes that never succeed:

Your health check endpoint returns 503 during startup, so Kubernetes never routes traffic to your pod.

// Bad health check - always returns ready
app.get('/health', (req, res) => {
  res.status(200).json({ status: 'ok' });
});

// Better health check - actually checks if app is ready
let isReady = false;

// Set ready after database connection, etc.
connectToDatabase().then(() => {
  isReady = true;
});

app.get('/health', (req, res) => {
  if (isReady) {
    res.status(200).json({ status: 'ready' });
  } else {
    res.status(503).json({ status: 'starting' });
  }
});

Networking Issues That Will Drive You Insane

Service can't reach other services:

DNS resolution in Kubernetes is weird. Services can reach each other by name within the same namespace, but cross-namespace requires FQDN.

## This works within the same namespace
DATABASE_URL: postgres://user:pass@postgres:5432/db

## This works across namespaces
DATABASE_URL: postgres://user:pass@postgres.database.svc.cluster.local:5432/db

Ingress routing that just doesn't work:

Your ingress looks correct but requests get 404 or route to the wrong service.

## Make sure your service selector matches pod labels
apiVersion: v1
kind: Service
spec:
  selector:
    app: your-app          # Must match pod labels
  ports:
  - port: 80
    targetPort: 3000       # Must match container port

---
## And your ingress path must match service
apiVersion: networking.k8s.io/v1
kind: Ingress
spec:
  rules:
  - http:
      paths:
      - path: /api
        pathType: Prefix
        backend:
          service:
            name: your-app  # Must match service name
            port:
              number: 80   # Must match service port

ConfigMap and Secret Mount Issues

Your environment variables just aren't there:

## Wrong way - typos in key names kill you
env:
- name: DATABASE_URL
  valueFrom:
    secretKeyRef:
      name: app-secrets
      key: database-url    # Typo! Should be database_url

## Right way - check the actual secret keys first

## Debug ConfigMaps and Secrets
kubectl get secret app-secrets -o yaml
kubectl describe configmap app-config

File mounts that don't show up:

Your config file should be mounted at /etc/config/app.yml but the directory is empty.

## Check mount path vs container path
volumeMounts:
- name: config
  mountPath: /etc/config          # Directory, not file
  subPath: app.yml               # Specific file from configmap

volumes:
- name: config
  configMap:
    name: app-config
    items:
    - key: app.yml               # Key in configmap
      path: app.yml              # File name in mount

The Nuclear Debug Options for Kubernetes

When you're truly desperate:

## Create a debug pod in the same network namespace
kubectl run debug --image=busybox -it --rm --restart=Never -- /bin/sh

## Or attach to running pod
kubectl exec -it your-app-xxx -- /bin/bash

## Port forward to access services locally
kubectl port-forward service/your-app 8080:80

## Check what's actually running in the container
kubectl exec your-app-xxx -- ps aux
kubectl exec your-app-xxx -- netstat -tlnp
kubectl exec your-app-xxx -- env

Copy files from failing pods:

## Get config files to see what's actually mounted
kubectl cp your-app-xxx:/etc/config ./debug-config

## Get logs from filesystem if kubectl logs doesn't work
kubectl cp your-app-xxx:/var/log ./debug-logs

Docker Registry Authentication Nightmares

ImagePullBackOff with private registries:

The image exists, you can pull it locally, but Kubernetes can't pull it.

## Create registry secret
kubectl create secret docker-registry myregistrykey \
  --docker-server=my-registry.com \
  --docker-username=myuser \
  --docker-password=mypassword

## Check if the secret is actually correct
kubectl get secret myregistrykey -o yaml | base64 -d

For AWS ECR:

## ECR tokens expire every 12 hours
aws ecr get-login-password --region us-west-2 | \
  docker login --username AWS --password-stdin 123456789012.dkr.ecr.us-west-2.amazonaws.com

## Set up automatic token refresh in your pipeline

Container Startup Failures

Exit code 125: Docker daemon error
Usually a problem with the Docker image or runtime configuration.

Exit code 126: Container command not executable
The command you specified in CMD or ENTRYPOINT doesn't have execute permissions.

## Fix permissions in Dockerfile
COPY script.sh /app/
RUN chmod +x /app/script.sh
CMD ["/app/script.sh"]

Exit code 127: Container command not found
The command doesn't exist in the container.

## Make sure the command exists
RUN which node  # Check if node is available
CMD ["node", "app.js"]

The key insight: containers fail fast and with limited context. Enable verbose logging, check all your assumptions, and remember that what works in your terminal might not work in a minimal container environment.