KEDA - Kubernetes Event-driven Autoscaling

KEDA (Kubernetes Event-driven Autoscaler) is a CNCF graduated project originally created by Microsoft and Red Hat. The latest version is v2.17.2, released in June 2024, and it fixes Kubernetes' biggest autoscaling fuckup: scaling based on metrics that actually matter.

The Problem: CPU Metrics Are Bullshit

Traditional Kubernetes Horizontal Pod Autoscaler (HPA) only looks at CPU and memory. That's like judging a restaurant's popularity by how hot the kitchen gets. Your message queue could have 10,000 pending jobs, but if your workers aren't pegging CPU, HPA doesn't give a shit.

I learned this the hard way when our Redis queue went completely nuts - had to be 40-something thousand messages, maybe 50k? Hard to say exactly - and HPA just sat there like a useless brick while our response times turned to absolute shit. CPU was fine, so why scale?

How KEDA Actually Works

KEDA has three components that don't suck:

KEDA Operator monitors external stuff - your message queues, databases, whatever. It creates ScaledObjects and ScaledJobs that actually make sense for your workload.

Metrics Server translates external metrics into something Kubernetes HPA can understand. It's basically the middleware that makes KEDA play nice with existing K8s autoscaling.

Scalers connect to 60+ external services including Apache Kafka, Redis, RabbitMQ, AWS SQS, Azure Service Bus, Prometheus, and tons more.

Scale-to-Zero: Actually Useful

KEDA can scale your pods to zero when there's no work. Not "minimum 1 replica" - actual zero. When messages show up in your queue, it spins up pods in about 30 seconds (don't believe the "within seconds" marketing BS).

This saved us maybe 60% on our staging environment costs, hard to say exactly. Production? That's where you learn the hard way that your startup time is actually like 45 seconds on a bad day, not the 5 seconds you thought, and users start bitching about timeouts.

Real Production Gotchas Nobody Tells You

KEDA operator resource usage: The docs say 200MB RAM but that's bullshit. Plan for at least 400-500MB RAM, probably more if your cluster decides to be an asshole. Each scaler hammers your APIs every 30 seconds - hope you like those sweet, sweet cloud provider API bills.

Scale-to-zero timing: That "within seconds" claim? Complete bullshit. Expect like 30-60 seconds for the first pod, maybe longer if your image is huge. If you need sub-5-second response times, scale-to-zero will piss off your users.

Event source failures: If your Redis/Kafka/whatever goes down, KEDA keeps your app at current scale. It doesn't freak out and scale to zero, which is actually pretty smart.

Authentication debugging: TriggerAuthentication fails silently like a passive-aggressive coworker who leaves you Post-it notes about your failures instead of just telling you. You'll spend 6 hours debugging why your ScaledObject sits there doing absolutely nothing, only to discover you fat-fingered the secret name. Again. Always check kubectl logs -l app=keda-operator -n keda first, not after you've already questioned your career choices.

When KEDA Actually Makes Sense

• Event-driven workloads (message queues, batch processing)
• Variable traffic patterns where CPU scaling is useless
• Cost-sensitive environments where scale-to-zero matters
• Integration with cloud services that HPA can't see

KEDA works with Deployments, StatefulSets, and Jobs, plus any custom resource that implements the /scale sub-resource. It plays nice with existing VPA and other Kubernetes tools.

Anyway, here's how KEDA compares to the other autoscaling options that probably aren't working for you either.

I've deployed KEDA about a dozen times across different teams. Here's what actually works and what will waste your time.

Installation: Just Use Helm

Skip the YAML manifests unless you love pain. The official Helm chart handles all the CRD and certificate BS for you:

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

Takes about 2 minutes if your cluster doesn't hate you. The YAML manifests exist but you'll spend like 3 hours dealing with webhook certificates, RBAC bullshit, and admission controllers that some asshole forgot to document.

OpenShift users: Use OperatorHub instead. It handles the security context constraints that will otherwise make your life miserable.

ScaledObject: Where Most People Screw Up

Here's a Redis scaler that actually works in production:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: worker-scaler
spec:
  scaleTargetRef:
    name: worker-deployment
  minReplicaCount: 1    # Don't set to 0 in prod unless you like angry users
  maxReplicaCount: 20   # Set this or prepare for chaos
  triggers:
  - type: redis
    metadata:
      address: redis.default.svc.cluster.local:6379
      listName: work-queue
      listLength: '5'
    authenticationRef:
      name: redis-auth

Common fuckups:

• Setting minReplicaCount: 0 in production without thinking about cold start times
• Forgetting maxReplicaCount and watching your cluster explode
• Using redis-server:6379 instead of the full Kubernetes service name
• Not setting up TriggerAuthentication for anything that needs auth

Authentication: The Part That Always Breaks

TriggerAuthentication fails silently. When your ScaledObject isn't scaling, this is usually why:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: redis-auth
spec:
  secretTargetRef:
  - parameter: password
    name: redis-secret
    key: password

Debug with: kubectl logs -l app=keda-operator -n keda | grep -i auth

Pro tip: Test your auth separately before adding it to ScaledObject. I've seen teams spend fucking days debugging scaling when the problem was just a typo in the secret name.

Scalers That Actually Work Well

Apache Kafka: Solid. Scales based on consumer lag. Works great for event processing.

RabbitMQ: Reliable. Queue depth scaling works exactly as expected. Just don't forget the management plugin.

AWS SQS: Works well with IRSA. Approximate message count is good enough for most use cases.

Prometheus: Powerful but debugging PromQL queries in KEDA will make you hate life. Test your queries in Prometheus first.

Cron: Perfect for predictable workloads. We use it to pre-scale before our morning batch jobs.

Production Reality Check

Here's the shit nobody tells you in the marketing demos:

Resource requirements: The docs say 200MB RAM, 100m CPU. That's a fucking joke. We're using way more than that, probably 400-500MB with around 20 ScaledObjects. Hit some crazy memory spike when everyone deployed at once - think it was like 700-800MB? That was a fun Friday. Some genius created like 200 ScaledObjects all polling every 30 seconds. Each scaler hammers your APIs constantly - hope you like those sweet, sweet AWS CloudWatch API calls at $0.01 per 1,000 requests.

Scaling storms: Someone's query was scanning the entire user table every 30 seconds. Took forever to figure out which one was doing it. Database completely shit the bed, took like 2 hours to recover, and my phone wouldn't stop buzzing. Turns out some dev forgot a WHERE clause in their monitoring script. Always test your queries under load, preferably not in production like we did.

KEDA operator restarts: When KEDA crashes (and it will crash - we've seen it die from memory pressure, certificate issues, and random network timeouts), all scaling just stops. Your apps keep running at whatever scale they were at, which is great until your morning batch jobs don't scale up and your CEO asks why reports are 3 hours late.

Scale-to-zero gotchas:

• Pod startup time matters. If your app takes 30+ seconds to start, users will notice.
• PersistentVolumeClaims don't get cleaned up automatically with ScaledJobs
• Some cloud load balancers freak out when target groups go to zero

Monitoring You Actually Need

Prometheus integration is essential. Watch these metrics:

• keda_scaler_errors_total - When your scalers are failing
• keda_scaled_object_paused - When scaling is broken
• keda_metrics_server_* - API server health

Set up alerts for scaler errors. Silent failures are KEDA's specialty - it'll fail quietly while your pods sit there doing nothing and your users get pissed.

When KEDA Isn't Worth It

• Simple web apps with predictable traffic - just use HPA
• Stateful workloads that can't handle restarts
• Real-time systems that need sub-second response times
• Teams that don't understand their event sources

KEDA is great for event-driven stuff. For everything else, it's probably overkill.

Still have questions? Of course you do - KEDA is powerful but not always intuitive. Here are the questions everyone asks when they're trying to figure out if KEDA is right for their setup.