KrakenD Production Troubleshooting - Fix the 3AM Problems

After dealing with production KrakenD deployments for years, these are the problems that wake you up at 3am. Most of them have simple fixes once you know what to look for.

Memory Leaks and OOM Kills

Symptom: KrakenD containers randomly restart, kubectl logs shows exit code 137, Kubernetes keeps killing your pods.

This usually hits when you're handling concurrent requests without proper resource limits. KrakenD spawns goroutines like crazy during traffic spikes, and if you don't configure memory limits correctly, Kubernetes will murder your pods. Check the Kubernetes deployment guide and Docker deployment best practices for proper resource configuration.

The fix that actually works:

resources:
  limits:
    memory: "1Gi"
    cpu: "1000m"
  requests:
    memory: "512Mi"
    cpu: "100m"

Set explicit memory limits in your Kubernetes deployment. Don't trust the defaults. I've seen teams lose entire weekends because they skipped this basic step. Also check your circuit breaker configuration - aggressive timeouts can cause request pileups. For monitoring, set up Prometheus metrics and Grafana dashboards to catch these issues early.

Configuration Validation Hell

Symptom: KrakenD starts but endpoints return 404s, config changes don't take effect, mysterious routing issues.

The number one cause: your JSON config has subtle syntax errors that don't break startup but break routing. KrakenD's error messages for config issues are... not great. You'll spend hours debugging what should be a 30-second fix. Use the configuration check command and configuration audit tool to catch these before deployment.

Check this first:

## Always validate before deploying
krakend check --config krakend.json

Common gotchas:

• Missing http:// in backend URLs (this one cost me 2 hours last month)
• Trailing slashes in endpoint paths - /api/users/ vs /api/users are different endpoints
• Wrong url_pattern vs endpoint matching - read the docs because this trips everyone up. Also check backend configuration and parameter forwarding guide.

Backend Service Discovery Failures

Symptom: 502 errors, "connection refused", KrakenD can't reach your services even though they're running.

In Kubernetes, this is usually DNS resolution. KrakenD tries to connect to backend-service:8080 but can't resolve the hostname because of namespace issues or service naming problems. Check the service discovery documentation and Kubernetes networking guide.

Debug the networking:

## From inside your KrakenD pod
kubectl exec -it krakend-pod -- nslookup backend-service
kubectl exec -it krakend-pod -- curl your-backend-service:8080/health
## Replace 'your-backend-service' with your actual service name and add http://

Kubernetes-specific fixes:

• Use full service names: backend-service.namespace.svc.cluster.local:8080
• Check your service selector labels match your backend pods
• Verify port numbers - common mistake is exposing 80 but backend listens on 8080

JWT Validation Breaking Authentication

Symptom: 401 errors for valid tokens, authentication works sporadically, users getting logged out randomly.

KrakenD's JWT validation is picky about token format and timing. Clock drift between services can cause valid tokens to be rejected as expired.

Common authentication failures:

• JWK endpoint unreachable - KrakenD can't fetch public keys
• Token algorithm mismatch - your auth service uses RS256, config says HS256
• Clock synchronization issues - NTP drift causes timing validation failures

The nuclear option that usually works:

{
  "auth/validator": {
    "alg": "RS256",
    "jwk_url": "https://your-auth.com/.well-known/jwks.json",
    "cache_ttl": "15m",
    "disable_jwk_security": true
  }
}

disable_jwk_security is a temporary hack while you fix the real issue. Don't leave it in production.

Rate Limiting Gone Wrong

Symptom: Legitimate traffic getting 429 errors, rate limits triggering during normal load, users complaining about blocked requests.

KrakenD's rate limiting configuration is confusing and the defaults are aggressive. Token bucket settings don't behave like most people expect.

Rate limiting that doesn't suck:

{
  "qos/ratelimit/token-bucket": {
    "max_rate": 1000,
    "capacity": 1000,
    "every": "1s"
  }
}

Start with generous limits and tune down based on actual traffic patterns. Monitor your rate limiting metrics in Grafana to see what's actually being blocked.

Production Monitoring That Actually Helps

When KrakenD breaks in production, you need monitoring that tells you what's wrong instead of just that something is wrong.

Most teams set up the basic metrics but miss the ones that matter during incidents. Check the telemetry overview and OpenTelemetry implementation guide for comprehensive monitoring setup.### Essential Metrics for Production KrakenD**Memory and CPU are basic

• these are the ones that save your ass:**Response time percentiles by endpoint:

P95 and P99 latencies tell you which endpoints are struggling before users complain. Monitor these per endpoint, not just globally. Set up Prometheus monitoring and Grafana dashboards for visualization.Circuit breaker state changes:

When circuit breakers start opening, you have maybe 60 seconds before the incident escalates. Set alerts on circuit breaker state transitions, not just error rates.Backend connection failures: Track connection refused, timeout, and DNS resolution errors separately.

Each indicates a different type of problem requiring different fixes.JWT validation failures: Separate auth failures from other 401s.

Clock drift and JWK endpoint issues show up here first.Request queue depth: KrakenD queues requests during backend slowdowns.

Queue depth growing means you're about to have a bad time.### Logging Configuration for TroubleshootingStandard KrakenD logs are useless for production debugging. You need structured logging with the right log levels. Check the logging documentation and Graylog integration guide for structured logging setup.json{"extra_config": {"telemetry/logging": {"level": "INFO","prefix": "[KRAKEND]","syslog": false,"stdout": true,"format": "json"}}}Critical log patterns to alert on:

• connection refused
• Backend connectivity issues
• context deadline exceeded
• Timeout problems
• jwt validation failed
• Authentication problems
• circuit breaker is open
• Backend failures### Health Check StrategyKrakenD's health endpoint only tells you if the process is running, not if it's working correctly.Better health checks test actual functionality:json{"endpoints": [{"endpoint": "/__health_detailed","method": "GET","backend": [{"url_pattern": "/health","host": ["http://critical-backend:8080"]}]}]}Create a health endpoint that actually tests backend connectivity.

Use this for Kubernetes liveness probes instead of the default health endpoint.### Alert Thresholds That WorkToo many false positives train people to ignore alerts. These thresholds are based on real production experience:

• Error rate > 1% for more than 2 minutes (not just a spike)
• P95 response time > 2x baseline for 5 minutes
• Circuit breaker open for any duration (immediate alert)
• Memory usage > 80% for 10 minutes (gives you time to scale before OOM kills)
• Backend connection failures > 10/minute for any backend### Performance Troubleshooting Workflow

When KrakenD performance goes to shit, follow this debugging sequence:**1.

Check backend health first**: 80% of KrakenD performance issues are actually backend issues.

Look at backend response times and error rates before diving into gateway metrics.2. Review traffic patterns: Sudden traffic spikes break things in predictable ways.

Check if the performance degradation correlates with traffic increases.3. Examine resource utilization: Memory and CPU spikes indicate resource constraints.

Scale horizontally before trying to tune configuration.4. Validate configuration changes: Recent config deployments cause most production issues.

Compare current config with the last known good configuration.5. Analyze request flows: Use distributed tracing to understand where requests are spending time.

Request aggregation and data manipulation can be expensive. Also check Zipkin integration and AWS X-Ray setup for tracing.### Capacity Planning Reality CheckKrakenD scales differently than other services. Most teams underestimate memory requirements and overestimate CPU needs.

Check the server dimensioning guide and clustering documentation for scaling best practices.Memory scaling:

Kraken

D uses roughly 100MB base + (concurrent_requests × 1MB) per endpoint. If you have 10 endpoints configured for 100 concurrent requests each, expect 1GB+ memory usage under load.CPU scaling: KrakenD is CPU efficient until you hit network I/O limits.

You'll usually need more replicas for network bandwidth before you need more CPU cores.Network bandwidth: API gateways push a lot of data.

Monitor network utilization

• many "performance" issues are actually bandwidth limitations. For detailed performance analysis, check the benchmarking documentation and performance testing guides.