KubeCost - Finally Know Where Your K8s Money Goes

Every month it's the same shit: AWS bill comes in 50% higher than expected and nobody knows why. Your EKS cluster is burning through money but AWS Cost Explorer just shows you EC2 instance costs. Useless.

The \"My Bill Exploded\" Problem

Here's what actually happens: Your data team spins up a Jupyter notebook for some "quick analysis" and accidentally leaves a model training job running all weekend. Meanwhile, your dev teams are deploying pods with CPU requests of 2 cores that actually use 0.1 cores. Nobody notices until the monthly AWS bill shows up at $47k instead of $30k.

Real production horror stories:

• ML team's TensorFlow job ran on GPU nodes for 3 weeks straight - cost $12k
• Staging namespace using production-sized RDS instances - nobody turned it off
• Persistent volumes growing to 500GB because log rotation was broken
• LoadBalancer services left running in 12 different namespaces

KubeCost Shows You Where the Money Goes

This is where KubeCost saves your sanity (and budget). Instead of staring at meaningless EC2 line items, you get actual answers.

KubeCost deploys as a pod in your cluster and scrapes Prometheus metrics to figure out exactly what's consuming resources. It then applies real AWS pricing data to show you costs down to the individual pod level.

The architecture is straightforward: KubeCost runs in your cluster, connects to Prometheus for resource metrics, pulls pricing data from cloud providers, and serves a web UI for cost visibility.

What you actually get:

• Pod-level costs: "That Redis pod costs $47/month"
• Namespace breakdown: "QA environment is costing $8k/month"
• Idle resource detection: "You have $3k/month of unused CPU"
• Network costs that AWS hides in separate line items

Works with GKE, AKS, and even on-premises clusters if you hate yourself enough to run K8s on bare metal.

IBM Bought Them (September 2024)

IBM acquired KubeCost for their Apptio portfolio. Good news: Enterprise features actually work now. Bad news: Expect pricing to go up. The free tier still exists (up to 250 CPU cores) but enterprise features cost real money.

What changed post-acquisition:

• Better multi-cluster federation
• Actual support instead of "check our Slack"
• Integration with IBM's FinOps tools
• More reliable RBAC integration
• KubeCost 2.7 (April 2025) added enhanced cost visibility and reporting flexibility
• Free trial of Enterprise Cloud extended through the rest of 2025

Links you'll actually need:

• Official installation guide (use this, not the old docs)
• Helm chart repo (where the real config lives)
• AWS EKS integration docs (AWS finally documented it properly)
• Community Slack (for when things break at 3am)
• GitHub issues (where the real problems are documented)
• r/kubernetes cost management thread (real user experiences)
• Cost optimization best practices (actually useful, not marketing fluff)

Let's talk about what happens when you actually install this thing in production. The marketing says "5 minutes" but plan for 2 hours minimum because something will definitely break. Note: KubeCost 2.7 (current version as of September 2025) is much more stable than earlier 2.x releases.

Installation Reality Check

The Basic Install:

helm repo add kubecost https://kubecost.github.io/cost-analyzer/
helm install kubecost kubecost/cost-analyzer -n kubecost --create-namespace

What actually happens:

1. Prometheus starts scraping and immediately runs out of memory
2. RBAC permissions are wrong and pods can't read cluster metrics
3. LoadBalancer service gets stuck in "Pending" because your cluster doesn't have one
4. Cost data shows up as $0 for everything because cloud pricing API calls are failing

Prometheus Memory Explosions

The "bundled Prometheus" is a lie. It's a minimal config that works for demo clusters but dies on real workloads. Here's what you need to know:

Resource requirements (real numbers):

• Small cluster (< 100 pods): 4GB RAM minimum, plan for 8GB
• Medium cluster (100-500 pods): 8GB RAM, 4 CPU cores
• Large cluster (500+ pods): 16GB+ RAM, dedicated nodes

Common failure modes:

• Prometheus OOMKilled after 24 hours
• Scraping timeouts causing missing metrics gaps
• Retention cleanup failing and eating all disk space
• Remote write to AWS Managed Prometheus timing out

Fix: Use existing Prometheus or size it properly

prometheus:
  server:
    resources:
      requests:
        memory: 8Gi
        cpu: 2000m
      limits:
        memory: 16Gi
    retention: "7d"
    storagePath: /data
    persistentVolume:
      size: 100Gi

Storage Grows Faster Than Expected

The "1GB per 1000 pods per month" estimate is bullshit. Plan for 3-5x that number.

What actually happens:

• Metrics data includes network topology that's huge
• PV cost tracking stores every volume resize event
• ETL aggregation creates duplicate data during multi-cluster setups
• Grafana dashboards query high-cardinality metrics and explode storage

Storage reality check (from production):

• 200-pod cluster: 5-10GB/month
• 1000-pod cluster: 50GB+/month
• Multi-cluster federation: 2-3x single cluster storage needs

Cost Data Accuracy Problems

Why your numbers don't match AWS bills:

1. Reserved Instance allocation is broken - KubeCost doesn't properly distribute RI discounts across workloads
2. Network costs are estimated - AWS Data Transfer pricing is complex and KubeCost guesses wrong
3. Spot instance pricing lags - Real spot prices change every 5 minutes, KubeCost updates hourly
4. EBS volume costs are weird - gp3 IOPS pricing isn't handled correctly in older versions

Bill reconciliation fixes most issues but requires:

• AWS Cost and Usage Reports configured properly
• S3 bucket with proper IAM permissions
• 24-48 hour delay for reconciled data to appear

Multi-Cluster Federation (Enterprise)

The federated ETL setup is where things get really fun. IBM's docs make it sound simple but here's the reality:

What breaks:

• ETL pipeline fails silently when one cluster has connectivity issues
• Data deduplication doesn't work with different cluster naming schemes
• Thanos integration requires custom Prometheus configs
• Cross-cluster networking policies block federation traffic

Network requirements nobody tells you:

• Clusters need service mesh connectivity or VPC peering
• Federation queries timeout on clusters >1000 nodes
• mTLS certificates expire and break federation silently

ARM64 Node Issues

Running KubeCost on ARM-based nodes (AWS Graviton, Apple Silicon) is broken in subtle ways:

Problems you'll hit:

• Cost-analyzer pod crashes on ARM nodes with "exec format error"
• Multi-arch images exist but Helm chart doesn't use them by default
• Prometheus node-exporter metrics are missing CPU topology data
• Network cost calculation fails on Graviton instances

Workaround: Pin KubeCost pods to AMD64 nodes:

nodeSelector:
  kubernetes.io/arch: amd64

Performance at Scale

Memory usage scales badly:

• 100 nodes: 2GB
• 500 nodes: 8GB
• 1000 nodes: 20GB+ (not the documented 4GB)
• 2000+ nodes: You need the enterprise federated architecture or it falls over

Query timeouts become common:

• UI becomes unusable with >30 days of retention
• API queries timeout after 2 minutes
• Grafana dashboard panels show "no data" intermittently

Real production configs:

• High availability setup requires 3+ replicas and shared storage
• Database backend (PostgreSQL) is mandatory for >500 nodes
• Redis caching required for acceptable UI performance

After dealing with all these deployment issues, you might wonder if there are better alternatives. That's fair - KubeCost isn't the only game in town, and depending on your situation, something else might make more sense.

Useful troubleshooting links:

• KubeCost memory issues GitHub thread
• Prometheus scaling guide
• AWS EKS KubeCost troubleshooting
• Community troubleshooting wiki
• IBM support portal (if you're paying for enterprise)
• KubeCost Slack debugging channel
• Stack Overflow: KubeCost deployment issues
• KubeCost performance tuning guide
• Helm chart configuration reference