OpenCost - Stop Getting Fucked by Mystery Kubernetes Bills

I've been running Kubernetes clusters for six years, and I can tell you that cost monitoring isn't optional anymore. It's the difference between explaining a reasonable infrastructure budget and having a very uncomfortable conversation with finance about why your "small microservices experiment" costs more than a Tesla.

The $50K Surprise Bill Problem

Here's what happens without cost monitoring: Someone spins up a "quick test" with GPU nodes, forgets about it over the weekend, and Monday morning you're staring at surprise AWS bills that can hit five figures. I've seen this exact scenario destroy entire Q4 budgets. Cost optimization studies show 32% of cloud spending is wasted, and Kubernetes makes visibility even worse.

Kubernetes makes this worse because traditional cloud billing shows you're paying for "EC2 instances" but doesn't tell you that your ChatGPT API wrapper is using 80% of your cluster resources. AWS Cost Explorer is useless for K8s workloads - it'll show you node costs but not which pods are burning money. The same problem exists with Azure Cost Management and Google Cloud Billing - they're great at showing you VM costs but completely blind to application-level resource consumption.

OpenCost Actually Works (Most of the Time)

OpenCost started as the open source guts of Kubecost before becoming its own CNCF project in June 2022 and advancing to CNCF Incubating status in October 2024. The core difference: it tracks cost allocation down to individual containers using actual Kubernetes resource requests and limits.

The magic happens because OpenCost integrates with cloud billing APIs to get real pricing data - reserved instances, spot pricing, enterprise discounts, all of it. No more bullshit static pricing models that are wrong by 40%. This is crucial when AWS spot instances can be 90% cheaper than on-demand, or when Azure Reserved Instances provide up to 72% savings that traditional tools completely miss.

What Actually Gets Tracked

• Container-level costs: Finally know which pod is eating your budget
• GPU costs: Critical if you're doing any ML work (spoiler: NVIDIA A100s cost $3.06/hour and GPUs are expensive as fuck)
• Persistent volume costs: That 10TB database you forgot about shows up here, including AWS EBS costs and persistent disk fees
• Network costs: Load balancer and data transfer fees that AWS loves to hide, plus NAT gateway costs that can surprise you
• Out-of-cluster resources: RDS, S3, anything your apps actually use via custom cost sources

The allocation model is pretty solid - it proportionally distributes node costs based on resource requests, which means if you're not setting requests/limits properly, your cost data will be garbage. But you should be setting those anyway if you're not a complete amateur. This follows the Kubernetes resource model and aligns with FinOps best practices for cloud cost allocation.

Production Reality Check

Version 1.117.3 is the latest as of September 2024. It's been stable in my production clusters for over a year. The main gotcha is Prometheus integration - if your Prometheus is fucked, OpenCost will be fucked too.

The RBAC permissions need read access to basically everything, which makes security teams nervous. But that's the price of actually knowing where your money goes. Check the security considerations and OpenSSF scorecard if your compliance team needs convincing.

Works with Kubernetes 1.21+ but honestly, if you're running anything older than 1.26 in production, cost monitoring is the least of your problems. The latest versions support Kubernetes 1.30 and integrate well with modern CNI plugins for accurate network cost tracking.

Convinced OpenCost is worth trying? Good. Now comes the fun part - actually getting it running. This is where most people discover that "just deploy with Helm" is never as simple as the docs make it sound.

The Prometheus Dependency Hell

First reality check: OpenCost needs Prometheus and if your Prometheus setup is janky, OpenCost will be janky too. I learned this the hard way when our cost data was missing random 6-hour chunks because Prometheus was running out of disk space. The official Prometheus documentation recommends monitoring disk usage, and the kube-prometheus-stack Helm chart includes good defaults for production.

The system requirements say 100m CPU and 256Mi memory, but that's bullshit if you have more than 50 nodes. Plan for at least 500m CPU and 1Gi memory for anything resembling production scale. Check the Kubernetes resource recommendations and resource quotas guide for sizing guidance. Also, make sure your Prometheus retention is set properly - 15 days minimum or you'll lose cost history.

The Helm Install (Usually Works)

helm repo add opencost-charts https://opencost.github.io/opencost-helm-chart
helm repo update
helm install opencost opencost-charts/opencost --namespace opencost --create-namespace

This works about 90% of the time. When it doesn't, it's usually because:

1. RBAC permissions are fucked: Check your cluster-admin privileges and review Kubernetes RBAC documentation
2. Prometheus endpoint is wrong: Default assumes http://prometheus-server.prometheus.svc.cluster.local:80 - verify with service discovery troubleshooting
3. Network policies blocking scraping: OpenCost needs to reach Prometheus and kubelet - check network policy configuration

If the Helm install fails, don't be a hero - delete the namespace and start over:

kubectl delete namespace opencost

AWS Billing Integration (The Real Pain)

Getting AWS Cost and Usage Reports working is where most people give up. You need:

1. IAM role with billing permissions - not just EC2, the actual billing APIs
2. CUR reports enabled - this takes 24 hours to start generating data
3. S3 bucket properly configured - get the region wrong and it won't work

The IAM policy in the docs is actually correct, but AWS's billing permissions are weird. I've had setups where it took 3 days for permissions to properly propagate. Check the AWS IAM best practices and troubleshooting guide if you're having permission issues.

Pro tip: Test with aws ce get-cost-and-usage first using the AWS CLI Cost Explorer commands. If that doesn't work, OpenCost won't work either.

Azure Setup (Slightly Less Painful)

Azure configuration is more straightforward but still has gotchas:

• Service principal needs "Cost Management Reader" role - see Azure role assignments
• Must be assigned at the subscription level, not resource group - check Azure subscription management
• Takes 2-4 hours for cost data to start showing up

The Azure CLI commands in the docs actually work, which is more than I can say for most Azure documentation. Use az account show to verify subscription access.

GCP (Surprisingly Smooth)

GCP setup is the easiest of the three clouds. Enable the BigQuery billing export, create a service account with BigQuery Data Viewer role, and you're done. GCP's billing APIs are fast and reliable compared to AWS and Azure.

Accessing Your Data

## Port forward to the UI (because ingress is always broken initially)
kubectl port-forward --namespace opencost service/opencost 9003:9003

## Hit the API directly (useful for debugging)
curl \"localhost:9003/allocation/compute?window=7d\"

The kubectl cost plugin is actually pretty useful once installed via Krew:

kubectl krew install cost
kubectl cost namespace --window 7d

Common Fuckups and Fixes

Empty cost data: Usually means Prometheus isn't scraping kubelet metrics. Check your Prometheus configuration and make sure the kubernetes-cadvisor job is working.

Wrong cost allocations: You probably have pods without resource requests/limits. OpenCost can't allocate costs properly if Kubernetes doesn't know how much resources pods want. Review resource management best practices.

Slow UI: The web interface gets sluggish with large clusters. Use the API directly or implement your own dashboard with Grafana using the OpenCost Grafana dashboard.

Version conflicts: Always use the latest stable release - newer versions have better resource allocation and bug fixes. Check the upgrade guide for breaking changes.

The OpenCost community Slack is actually helpful when shit breaks. Much better than opening GitHub issues and waiting days for responses.

For more deployment help, check out the official deployment guide, Helm chart documentation, Docker installation guide, Kubernetes operator setup, and production deployment examples.