Why do I need both of these damn things?

Look, I get it. You already have [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) for autoscaling and now someone wants you to install kube-state-metrics too. Here's the deal: - **Metrics Server:** Shows resource usage (CPU/memory) for HPA/VPA - **kube-state-metrics:** Shows object states (why pods are failing, replica counts, etc.) You need both because Metrics Server won't tell you why your deployment is stuck at 2/3 replicas. That's what kube-state-metrics does. Trust me, install both and stop asking questions.

How much memory will this thing actually eat?

Forget the "250MB" bullshit in the docs. In production: - Small cluster (10-50 nodes): 300-500MB - Medium cluster (50-200 nodes): 500-800MB - Large cluster (200+ nodes): 800MB-1.5GB I started with the recommended limits and watched it get OOM killed constantly until I bumped memory to 1GB. [Performance docs](https://github.com/kubernetes/kube-state-metrics#resource-recommendation) are optimistic at best.

Can I stop it from monitoring everything?

Yes, thank God. Use these flags to avoid metric explosion: ```bash --resources=pods,deployments,services --namespaces=production,staging --metric-allowlist=kube_pod_status.*,kube_deployment_.* ``` The [metric filtering docs](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/cli-arguments.md#available-options) are actually useful for once.

How the hell do I get Prometheus to scrape this?

If you're using [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack), it's automatic. Otherwise, add this to your scrape config: ```yaml - job_name: kube-state-metrics static_configs: - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080'] ``` Port 8080 by default, port 8081 for telemetry metrics. Don't forget the telemetry - that's how you know if the thing is broken.

Is this secure enough for production?

It's read-only API access, runs as non-root, and the [Kubernetes community](https://github.com/kubernetes/community/tree/master/sig-instrumentation) maintains it. Security-wise it's fine. The real issue is RBAC. You'll need cluster-wide read permissions or very specific [role bindings](https://kubernetes.io/docs/reference/access-authn-authz/rbac/). If your security team freaks out about ClusterRole, you can scope it per-namespace but you lose cluster-wide visibility.

What happens when this breaks?

Your Prometheus scrapes fail and you lose real-time cluster state visibility. Historical data is fine, but you're flying blind on current issues. The good news: it's stateless. Just restart the pod and it reconnects to the API server immediately. I've had zero data loss from restarts in 2+ years of running this.

Why are my metrics missing?

90% of the time it's one of these: 1. **RBAC permissions:** Check the ClusterRole allows reading the resources you want 2. **API server connectivity:** Look at pod logs for connection errors 3. **Wrong Prometheus config:** Verify the scrape target and port 4. **Resource filtering:** You probably filtered out the metrics you want Run `kubectl port-forward` and hit the `/metrics` endpoint directly. If you see metrics there, it's a Prometheus config problem.

Will this scale to my massive cluster?

If you have 1000+ nodes or 10,000+ pods, you need [horizontal sharding](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/horizontal-sharding.md). It works but adds complexity. For most clusters, a single instance with proper resource limits handles 100-500 nodes fine. I've run it on 300-node clusters without sharding.

Can I monitor my custom CRDs?

Yeah, through [Custom Resource State](https://github.com/kubernetes/kube-state-metrics/blob/main/docs/customresourcestate-metrics.md) configuration. You define a YAML config mapping your CRD fields to metrics. It's useful for monitoring operators like [cert-manager](https://cert-manager.io/) or database operators. But expect to spend time figuring out the YAML syntax - the examples are minimal.

What's new in v2.17.0 that I actually care about?

The unscheduled pod tracking is huge - `kube_pod_unscheduled_time_seconds` finally shows you when pods are stuck in Pending for too long. I've waited years for this metric. The deletion timestamp metrics (`kube_deployment_deletion_timestamp`, etc.) help track cleanup operations. And the enhanced `reason` labels on deployment conditions make debugging failed rollouts way easier. **Performance note:** v2.17.0 also includes better memory management with automemlimit support, which helps prevent those random OOM kills in large clusters. **Visualization:** Most people use [Grafana dashboards](https://grafana.com/grafana/dashboards/?search=kube-state-metrics) to visualize this data. There are dozens of pre-built dashboards, though most are overcomplicated. Start with something simple and build from there. **Grafana Integration:** Grafana provides the visualization layer for your kube-state-metrics data, with dozens of pre-built dashboards available. Popular dashboard options include the [Kubernetes cluster monitoring dashboard](https://grafana.com/grafana/dashboards/15661-k8s-dashboard-en-20250125/) which provides cluster-level insights, and specialized dashboards for [workload monitoring](https://grafana.com/grafana/dashboards/22523-eks-dashboard/) that focus on pod and deployment health.

Currently viewing the AI version

Switch to human version

kube-state-metrics: AI-Optimized Implementation Guide

Core Function & Critical Value Proposition

Primary Purpose: Exposes Kubernetes API object states as Prometheus metrics for cluster health monitoring and debugging

Critical Problem Solved: Provides real-time visibility into Kubernetes object states (why deployments are stuck, which pods are failing, node conditions) that standard metrics-server cannot provide

Operational Reality: Without this tool, debugging cluster issues requires manual kubectl commands and guesswork about object states

Technical Specifications & Architecture

System Requirements & Resource Reality

Memory Requirements (Production):
- Small cluster (10-50 nodes): 300-500MB
- Medium cluster (50-200 nodes): 500-800MB
- Large cluster (200+ nodes): 800MB-1.5GB
Default 250MB limit WILL cause OOM kills in real clusters
CPU Usage: 100-200m typically sufficient
Network: Single persistent watch connection to API server (not constant polling)

Current Version Intelligence

Latest Stable: v2.17.0 (September 1, 2025)
Critical New Metrics:
- kube_pod_unscheduled_time_seconds - tracks pod scheduling delays
- kube_deployment_deletion_timestamp - monitors cleanup operations
- Enhanced reason labels for deployment condition debugging
Go Version: 1.24.6 with client-go v0.33.4
Compatibility: Match client-go version to avoid API compatibility failures

Critical Deployment Configurations

Production-Ready Helm Deployment

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install kube-state-metrics prometheus-community/kube-state-metrics

Required Configuration Overrides:

resources:
  requests:
    cpu: 100m
    memory: 500Mi
  limits:
    cpu: 200m
    memory: 1Gi
service:
  port: 8081  # Avoid 8080 conflicts
  targetPort: 8081
telemetryPort: 8081
telemetryHost: "0.0.0.0"

Critical Failure Points & Solutions

RBAC Permission Failures

Symptoms: Missing metrics, connection refused errors
Root Cause: ClusterRole permissions insufficient or Pod Security Standards blocking access
Solution: Ensure system:metrics access or custom policy for Pod Security Standards
Debug Command: kubectl port-forward to test direct metrics endpoint access

Memory & Resource Issues

OOM Kill Pattern: Occurs when cluster has 2000+ pods with default 250MB limit
Scaling Formula: ~400KB per 1000 pods + base overhead
Production Minimum: 500MB for any real cluster
Large Cluster Threshold: 1000+ nodes or 10,000+ pods requires horizontal sharding

Port Conflicts

Common Issue: Port 8080 conflicts with other services
Solution: Use port 8081 for both service and telemetry
Monitoring Ports: 8080/8081 for metrics, 8081 for telemetry/health

Comparative Analysis vs Alternatives

Tool	Purpose	Resource Usage	Reliability	Setup Complexity
kube-state-metrics	Object state visibility	200MB-800MB	High (stateless, reconnects automatically)	Medium (RBAC issues)
metrics-server	Resource usage for HPA	40MB (until OOM)	Medium (random OOM kills)	Low (usually pre-installed)
Prometheus Node Exporter	System-level metrics	20MB (stable)	Very High	Low
cAdvisor	Container resource usage	Kubelet overhead	Medium (breaks with Kubelet)	None (built-in)

Critical Production Warnings

What Will Break Your Deployment

Default Memory Limits: 250MB limit causes OOM kills in clusters with >500 pods
RBAC Scope: Cluster-wide read permissions required; namespace-scoped loses cluster visibility
API Server Connectivity: Single point of failure; connection issues = complete monitoring loss
Port Conflicts: Default 8080 conflicts with common services
Client-Go Version Mismatch: Causes API compatibility issues with specific Kubernetes versions

Cloud Platform Gotchas

GKE: Built-in version limited, sends to Cloud Monitoring (not Prometheus)
EKS: Not included by default; must install separately
AKS: Container Insights provides subset; install full version for complete metrics

Essential Monitoring & Health Checks

Critical Health Metrics

kube_state_metrics_list_total - should increment regularly (API connectivity)
kube_state_metrics_watch_total - tracks API watch connections
process_resident_memory_bytes - memory usage stability
up - basic service availability

Key Debugging Metrics

kube_pod_container_status_restarts_total - crashloop detection
kube_pod_status_phase + kube_pod_status_conditions - pending pod analysis
kube_deployment_status_replicas_available vs kube_deployment_spec_replicas - scaling issues
kube_job_status_failed + kube_job_status_succeeded - job failure patterns
kube_node_status_condition - node health before complete failure

Scaling & Performance Thresholds

Single Instance Limits

Maximum Recommended: 500 nodes, 5000 pods
Performance Degradation: Starts at 1000+ nodes
Hard Limits: 10,000+ pods requires sharding

Horizontal Sharding Requirements

Trigger Point: 1000+ nodes OR 10,000+ pods
Implementation: StatefulSet with autosharding examples
Complexity Cost: Debugging which instance monitors specific objects becomes difficult
Monitoring Requirement: Track health metrics per shard instance

Integration Requirements

Prometheus Configuration

- job_name: kube-state-metrics
  static_configs:
  - targets: ['kube-state-metrics.kube-system.svc.cluster.local:8080']

Resource Filtering (Large Clusters)

--resources=pods,deployments,services
--namespaces=production,staging
--metric-allowlist=kube_pod_status.*,kube_deployment_.*

Operational Intelligence

Time Investment Reality

Basic Setup: 30-60 minutes with Helm
RBAC Debugging: 2-4 hours for complex security policies
Large Cluster Sharding: 4-8 hours initial setup + ongoing complexity
Custom CRD Integration: 2-6 hours per CRD depending on complexity

Support & Community Quality

Maintenance: Official Kubernetes SIG Instrumentation (high quality)
Community: Active Kubernetes Slack channel with responsive help
Documentation: Good for basic setup, lacking operational details
Breaking Changes: Minimal; version upgrades generally safe

Migration & Breaking Points

Upgrade Path: Generally smooth; backward compatible metrics
Kubernetes Version Support: Follows client-go compatibility matrix
API Changes: Rarely break existing metrics; new metrics added regularly
Resource Impact: Memory usage grows linearly with cluster size

Decision Criteria

Deploy When:

Cluster has >50 pods or production workloads
Need visibility into deployment/pod state issues
Running Prometheus for monitoring
Debugging scaling or scheduling problems

Skip When:

Single-node development clusters
Only need resource usage metrics (use metrics-server)
Cloud provider monitoring sufficient for use case

Cost-Benefit Analysis

Resource Cost: 500MB-1GB RAM, 100-200m CPU
Operational Value: Eliminates manual kubectl debugging, provides early warning for cluster issues
Time Savings: 2-4 hours per incident avoided through proactive monitoring
Hidden Costs: RBAC complexity, potential port conflicts, scaling complexity for large clusters

Useful Links for Further Investigation

Resources That Don't Suck

Link	Description
GitHub Repository	The source of truth. Read the releases and issues before asking questions that have already been answered.
Official Kubernetes Docs	Basic overview that glosses over the hard parts, but covers the concepts.
Metrics Reference	Complete list of what metrics you get. Bookmark this - you'll reference it constantly.
CLI Arguments	How to configure filtering, sharding, and other options that actually matter.
Prometheus Community Helm Chart	Use this. Don't be a hero and write your own manifests.
Manual Manifests	If you can't use Helm, these work but you'll need to fix the resource limits.
Sharding Examples	For large clusters. The documentation here is actually decent.
Google GKE	Built-in but limited. Install your own if you want full functionality.
AWS EKS with Prometheus	AWS doesn't include this by default. Use their managed Prometheus or install yourself.
Azure AKS Monitoring	Container Insights has some kube-state-metrics data but not everything.
kube-prometheus-stack	Complete monitoring solution. This includes kube-state-metrics, Prometheus, Grafana, and Alertmanager. Just install this if you want everything to work together.
Prometheus Operator	If you want to manage Prometheus deployments at scale. More complex but powerful.
Grafana Dashboards	Pre-built dashboards. Some are good, most are overcomplicated. Start simple.
Kubernetes Slack #kube-state-metrics	Active community. People actually help here, but read the docs first or prepare for RTFM responses.
SIG Instrumentation	The team that maintains this. They know their shit.

kube-state-metrics: AI-Optimized Implementation Guide

Core Function & Critical Value Proposition

Technical Specifications & Architecture

System Requirements & Resource Reality

Current Version Intelligence

Critical Deployment Configurations

Production-Ready Helm Deployment

Critical Failure Points & Solutions

RBAC Permission Failures

Memory & Resource Issues

Port Conflicts

Comparative Analysis vs Alternatives

Critical Production Warnings

What Will Break Your Deployment

Cloud Platform Gotchas

Essential Monitoring & Health Checks

Critical Health Metrics

Key Debugging Metrics

Scaling & Performance Thresholds

Single Instance Limits

Horizontal Sharding Requirements

Integration Requirements

Prometheus Configuration

Resource Filtering (Large Clusters)

Operational Intelligence

Time Investment Reality

Support & Community Quality

Migration & Breaking Points

Decision Criteria

Cost-Benefit Analysis

Useful Links for Further Investigation

Resources That Don't Suck

Related Tools & Recommendations

Prometheus + Grafana + Jaeger: Stop Debugging Microservices Like It's 2015

GitOps Integration Hell: Docker + Kubernetes + ArgoCD + Prometheus

Kafka + MongoDB + Kubernetes + Prometheus Integration - When Event Streams Break

Set Up Microservices Monitoring That Actually Works

OpenTelemetry + Jaeger + Grafana on Kubernetes - The Stack That Actually Works

RAG on Kubernetes: Why You Probably Don't Need It (But If You Do, Here's How)

Grafana - The Monitoring Dashboard That Doesn't Suck

Fix Helm When It Inevitably Breaks - Debug Guide

Helm - Because Managing 47 YAML Files Will Drive You Insane

Making Pulumi, Kubernetes, Helm, and GitOps Actually Work Together

Google Pixel 10 Phones Launch with Triple Cameras and Tensor G5

Dutch Axelera AI Seeks €150M+ as Europe Bets on Chip Sovereignty

Setting Up Prometheus Monitoring That Won't Make You Hate Your Job

Alertmanager - Stop Getting 500 Alerts When One Server Dies

Datadog Cost Management - Stop Your Monitoring Bill From Destroying Your Budget

Datadog vs New Relic vs Sentry: Real Pricing Breakdown (From Someone Who's Actually Paid These Bills)

Datadog Enterprise Pricing - What It Actually Costs When Your Shit Breaks at 3AM

Samsung Wins 'Oscars of Innovation' for Revolutionary Cooling Tech

Nvidia's $45B Earnings Test: Beat Impossible Expectations or Watch Tech Crash

New Relic - Application Monitoring That Actually Works (If You Can Afford It)