Why Your Kubernetes Autoscaler Is Slow as Hell

Your cluster autoscaler takes forever to spin up nodes during traffic spikes. I've seen this problem enough times to recognize the patterns.

The Shit That's Actually Broken

Your scan interval is probably wrong. Most people leave it at the default 10 seconds, which sounds fast but production clusters usually bump it to 15-30 seconds to avoid hammering the API server. Problem is, that means your autoscaler might not even notice pending pods for 30+ seconds before starting the 5+ minute node provisioning process.

Had a traffic spike last year where the autoscaler just sat there doing nothing. Took way too long to realize what was happening - it wasn't even scanning for new work.

AWS will throttle you during spikes. Auto Scaling Groups have undisclosed API rate limits that you'll hit during any real traffic event. The autoscaler just fails silently when this happens. No errors, no warnings, just...nothing.

Spent a few hours debugging this during an incident before realizing AWS was just rate limiting every scaling request.

Too many node groups kills performance. I've seen clusters with 20+ node groups because someone thought they needed a separate group for every instance type. The autoscaler has to simulate every pending pod against every possible placement. With that many groups, it spends 30-60 seconds just thinking before doing anything.

The Config That Actually Matters

Forget the documentation - here's what works in production:

Scan intervals based on reality:

• Small clusters: --scan-interval=10s (if you're lucky)
• Medium clusters: --scan-interval=15s (still slow during emergencies)
• Large clusters: --scan-interval=20s (pray you don't need fast scaling)

Never go above 30 seconds unless you enjoy watching your app crash while pods stay pending.

Resource limits that don't suck:
The default 100MB memory limit is a joke. I've seen autoscaler pods OOM during large scale events, which is peak irony.

resources:
  requests:
    memory: \"1Gi\"    # Start here or suffer
    cpu: \"500m\"      # CPU-bound during simulation
  limits:
    memory: \"2Gi\"    # Give it room to breathe
    cpu: \"1\"         # More for large clusters

Node groups that make sense:
Stop creating a node group for every instance type. I consolidate down to 3-5 max:

• General compute (mixed instances)
• Memory-heavy (for your data hogs)
• GPU (if you're doing ML nonsense)
• Spot instances (cheap and disposable)

More groups = exponentially slower decisions. The simulation overhead will kill you.

The Weird Shit That Breaks Everything

Pod Disruption Budgets become simulation hell. Each PDB adds complexity to the scale-down calculations. I've seen clusters with tons of PDBs take 10+ minutes just to figure out if it's safe to remove a node.

One cluster I worked on had PDBs for everything, including services that could handle full outages. Removing unnecessary PDBs cut scale-down time significantly.

DaemonSets pile up and slow everything down. Every DaemonSet has to be considered during node operations. Enterprise clusters love their monitoring, security, and networking tools - I've seen 15+ DaemonSets that turned every node operation into a crawl.

Multi-zone setups create weird bottlenecks. AWS environments with 6+ availability zones see notable simulation overhead. Unless you have regulatory requirements, stick to 3 zones max.

The real fix isn't tweaking scan intervals - it's fixing your cluster architecture so the autoscaler has less shit to think about.

The difference between reading about autoscaler optimization and actually doing it in production is like the difference between reading about surgery and cutting someone open. Everything looks easy until you're elbow-deep in a cluster that's been frankensteined together over three years.

Start Here (do this first or you'll regret it)

Fix the scan interval first. This takes 30 seconds to change and saves you hours of pain later. Don't overthink it:

• Small clusters: --scan-interval=10s
• Medium clusters: --scan-interval=15s
• Large clusters: --scan-interval=20s

I've never seen a cluster that needed anything faster than 10s or slower than 20s. The documentation is usually wrong about this.

Give your autoscaler pod some goddamn memory. The default 100MB is embarrassing. I usually start with 1GB and scale up from there:

resources:
  requests:
    memory: "1Gi"    # Start here
    cpu: "500m"      # Simulation is CPU-bound
  limits:
    memory: "2Gi"    # Room for burst scaling
    cpu: "1"         # More for large clusters

Seen too many autoscaler pods OOM during scale events. Peak irony.

Switch to least-waste expander. One line change, immediate cost and performance benefits:

--expander=least-waste

The Architecture Surgery (plan for pain)

Consolidate your node groups. This is where things get political. Some architect three years ago created 20+ node groups because "flexibility" and now you get to explain why that was dumb.

I usually target 3-5 groups max:

• General compute (mixed instances, most workloads)
• Memory-heavy (data processing, caches)
• GPU (ML/AI workloads)
• Spot instances (fault-tolerant stuff)

Use mixed instance policies within each group instead of creating a group for every instance type. Your simulation overhead will thank you.

Node group redesign process (from painful experience):

1. Map your current workloads to see what actually needs special hardware
2. Consolidate gradually - don't do it all at once unless you enjoy outages
3. Test with non-critical workloads first
4. Have rollback plans because something will break

Cloud Provider Reality Checks

AWS: API rate limits will fuck you during traffic spikes. The requests per second limit sounds like a lot until peak traffic hits and everyone's fighting for the same quota.

--max-concurrent-scale-ups=5
--max-nodes-total=1000

GCP: Generally more reliable but has weird quota limits you won't discover until you hit them. Instance groups usually provision faster than AWS but the quotas are region/project specific.

Azure: VM Scale Sets are a dice roll. Sometimes they provision in under 2 minutes, sometimes they take 15+ with no clear pattern. I usually build in extra buffer time for Azure.

The Political Minefield (advanced fuckery)

Pod Disruption Budget cleanup. This is where you discover that someone put PDBs on stateless services "just in case." Each unnecessary PDB adds minutes to scale-down operations.

I've seen significant scale-down improvement just from removing PDBs that shouldn't exist. But you'll need to get buy-in from teams who think their stateless web service needs a PDB.

DaemonSet consolidation. Enterprise clusters collect DaemonSets like Pokemon cards. Monitoring, security, logging, networking - suddenly you have 15+ DaemonSets that slow every node operation.

Consolidation means getting security teams to accept fewer monitoring agents and operations teams to consolidate logging. Good luck with that.

Monitoring What Actually Matters

Most teams monitor everything except what breaks. Focus on these metrics:

• cluster_autoscaler_function_duration_seconds - if consistently above 5s, your autoscaler is struggling
• cluster_autoscaler_failed_scale_ups_total - non-zero means API limits or capacity issues
• cluster_autoscaler_nodes_count - trend analysis for capacity planning

Don't get lost in fancy dashboards. These three metrics tell you 90% of what you need to know.

The Brutal Truth

You can optimize scan intervals and resource limits all day, but if your cluster architecture is fundamentally broken (too many node groups, excessive PDBs, tons of DaemonSets), you're polishing a turd.

Fix the architecture first. Everything else is just configuration tweaks.

Most "autoscaler performance issues" are actually "cluster design issues" in disguise. The autoscaler is just the messenger getting shot for delivering bad news about your architecture choices.