kube-state-metrics (KSM) is an official Kubernetes add-on agent that listens to the Kubernetes API server and generates Prometheus-format metrics about the state of Kubernetes objects (such as Pods, Deployments, Nodes, StatefulSets, and CronJobs).
Unlike agent metrics collectors like cAdvisor or node_exporter, KSM does not measure resource usage (like CPU usage, memory consumption, or disk I/O). Instead, it translates the raw state of objects in the Kubernetes API into structured metrics.
Core Concept: Usage vs. State
cAdvisor / Node Exporter (Usage): "How much CPU is Pod X consuming right now?"
kube-state-metrics (State): "How many replicas are ready in Deployment Y?", "When did CronJob Z last succeed?", "What phase is Pod A in?"
How kube-state-metrics Exposes Metrics
┌─────────────────────────┐
│ Kubernetes API Server │
└────────────┬────────────┘
│ Watch / List (Informer Cache)
▼
┌─────────────────────────┐
│ kube-state-metrics │ ─── (Generates OpenMetrics in-memory)
└────────────┬────────────┘
│ HTTP GET /metrics (Port 8080)
▼
┌─────────────────────────┐
│ Prometheus Server │
└─────────────────────────┘
1. Consuming the API via Kubernetes Informers
KSM connects directly to the Kubernetes API server using standard Client-GO Informers.
- Instead of constantly polling the API server with heavy requests, KSM maintains an in-memory cache updated in real-time via long-polling watch streams.
- When a resource changes (e.g., a Deployment scales down or a Pod enters CrashLoopBackOff), the local cache updates instantly.
2. Generating Prometheus OpenMetrics In-Memory
KSM parses the fields of the Kubernetes object specs and status subresources (such as .status.phase, .spec.replicas, .status.conditions) and converts them directly into Prometheus gauge/counter metrics.
3. Serving via HTTP Endpoint
KSM hosts a lightweight HTTP server (default port :8080 at path /metrics).
It does not store metrics over time, nor does it push metrics out.
When a Prometheus server scrapes KSM's /metrics endpoint, KSM reads its in-memory snapshot, formats the raw text payload, and streams it back.
Key Technical Properties
Property
Details
Port & Path
Port 8080 (/metrics for cluster state), Port 8081 (/metrics for KSM internal performance telemetry)
Data Format
Prometheus text format / OpenMetrics standard
Metric Types
Predominantly Gauge metrics representing instantaneous state (1 or 0 state representations)
Scaling
Supports Horizontal Sharding and resource filtering (--resources, --namespaces) for large clusters
Example Metric Output
A scrape request to kube-state-metrics for a Deployment produces plain-text metrics like this:
# HELP kube_deployment_spec_replicas The desired number of pods declared in the deployment spec.
# TYPE kube_deployment_spec_replicas gauge
kube_deployment_spec_replicas{namespace="default",deployment="api-server"} 3
# HELP kube_deployment_status_replicas_available The number of available pods created by the deployment.
# TYPE kube_deployment_status_replicas_available gauge
kube_deployment_status_replicas_available{namespace="default",deployment="api-server"} 2
Do we need kube-state-metrics if we have Prometheus deployed in the cluster?
Yes, you still need kube-state-metrics (KSM) even if Prometheus is deployed in your cluster.
Prometheus is the engine that collects, stores, and queries time-series data, while kube-state-metrics is the agent that generates metrics about the state of Kubernetes API objects.
Prometheus does not natively inspect Kubernetes API objects on its own—it relies on exporters like KSM to expose that data.
What Prometheus collects out-of-the-box vs. with KSM?
Metric Source
Responsible Agent
Examples of Metrics Provided
Node / OS Metrics
node_exporter
Node CPU utilization, RAM usage, disk space, network traffic.
Container Usage
cAdvisor (built into kubelet)
Container CPU throttling, memory usage (RSS), network bytes per container.
Control Plane
kube-apiserver, etcd, coredns
API latency, request counts, etcd commit durations.
Kubernetes State
kube-state-metrics
Deployment replica counts, CronJob last success times, Pod restart counts, pending PVCs, ingress statuses, TLS certificate secret expiration.
What happens without kube-state-metrics?
If you run Prometheus without KSM, you lose visibility into the high-level health and configuration of your cluster resources. You will not be able to:
- Alert on Deployment Health: You won't know if spec.replicas (desired) doesn't match status.available_replicas.
- Track Job / CronJob Success: Metrics like kube_cronjob_status_last_successful_time or kube_job_status_failed won't exist.
- Monitor Pod Lifecycle States: You won't have metrics for Pods stuck in Pending, CrashLoopBackOff status codes, or ImagePullBackOff.
- Track Resource Requests vs. Limits: You won't be able to compare requested CPU/memory (kube_pod_container_resource_requests) against node capacity to measure cluster overcommit.
How They Work Together
┌─────────────────────────────────────────────────────────┐
│ Kubernetes API │
└────────────────────────────┬────────────────────────────┘
│ Watch API Objects
▼
┌───────────────────────┐
│ kube-state-metrics │
└───────────┬───────────┘
│ Exposes /metrics
▼
┌───────────────────────┐
│ Prometheus Server │ ◄── (Scrapes KSM)
└───────────────────────┘
- KSM listens to the API server and generates metrics representing resource states.
- Prometheus scrapes KSM's /metrics endpoint along with cAdvisor, node-exporter, and your application endpoints.
- Prometheus evaluates Alertmanager rules and stores the metrics for Grafana dashboards.
Standard Stack Setup
In standard Kubernetes monitoring deployments (such as the kube-prometheus-stack Helm chart or Prometheus Operator), kube-state-metrics is included by default as a core component alongside Prometheus and Alertmanager.
How to install kube-state-metrics?
kube-state-metrics (KSM) is typically deployed using one of four common approaches:
1. As Part of the kube-prometheus-stack (Most Common)
If you are setting up cluster monitoring with Prometheus and Grafana, kube-state-metrics is usually installed automatically as a bundled sub-chart.
Using the kube-prometheus-stack Helm chart:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install prometheus-stack prometheus-community/kube-prometheus-stack
kube-state-metrics runs out of the box with ServiceMonitors already configured.
2. Standalone Helm Chart
If you already have a Prometheus instance or another monitoring agent (like Datadog, New Relic, or Grafana Alloy) and just need KSM running, you can deploy the official standalone Helm chart:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-state-metrics prometheus-community/kube-state-metrics --namespace kube-system
3. Native Kustomize / Standard Manifests (kubectl apply)
For gitops workflows or lightweight clusters where Helm isn't used, you can apply the official manifests maintained directly in the upstream Git repository using Kustomize:
git clone https://github.com/kubernetes/kube-state-metrics.git
cd kube-state-metrics
kubectl apply -k examples/standard
Note: This creates the necessary ServiceAccount, ClusterRole, ClusterRoleBinding, Deployment, and Service in the kube-system namespace.
4. Cloud Managed Kubernetes Add-ons
On managed platforms (like GKE, EKS, or AKS), kube-state-metrics is often enabled as a one-click add-on or managed component integrated into cloud-native observability services (e.g., Google Cloud Observability, AWS CloudWatch Container Insights, or Azure Monitor).
What Gets Created in the Cluster?
Regardless of how you deploy it, KSM will create:
- ServiceAccount & ClusterRole/Binding: Grants read-only access (list, watch, get) to cluster API resources.
- Deployment: Runs the kube-state-metrics container.
- Service: Exposes the HTTP endpoint (usually on port 8080 at /metrics).
How to inspect which default metrics KSM exposes or how to disable unused ones?
Understanding which metrics kube-state-metrics (KSM) emits and how to prune unwanted metrics is key to controlling Prometheus ingestion volume and storage overhead.
Part 1: Inspecting Exposed Metrics
You can inspect the metrics exposed by KSM using three different methods:
1. Official Documentation Reference
The KSM GitHub repository contains generated documentation for every supported Kubernetes API group.
- Standard Docs: kube-state-metrics/docs contains dedicated files for each resource (e.g., pod-metrics.md, cronjob-metrics.md).
- Resource Status: The docs categorize metrics as STABLE (default), EXPERIMENTAL (alpha/beta features), or DEPRECATED.
2. Direct In-Cluster Scraping (curl / port-forward)
To see the exact payload KSM generates in your active cluster, port-forward to the KSM service and fetch /metrics:
# Port-forward the KSM service locally
kubectl port-forward svc/kube-state-metrics -n monitoring 8080:8080
# In a separate terminal, curl the metrics endpoint
curl http://localhost:8080/metrics
Filter for specific resource types using grep:
# View all Pod-related metrics exposed by KSM
curl -s http://localhost:8080/metrics | grep "^kube_pod_"
# View unique metric family names
curl -s http://localhost:8080/metrics | grep -v "^#" | cut -d'{' -f1 | sort -u
3. Prometheus Metric Explorer
In Prometheus, run the following query in the Expression Browser to list all distinct metric names harvested from the KSM job:
count by (__name__) ({job="kube-state-metrics"})
Part 2: Disabling or Filtering Unused Metrics
There are two primary ways to reduce metric volume: at the source (KSM Flags) or at the scraper (Prometheus Relabeling).
Method 1: Filtering at Source via KSM Flags (Recommended)
You can instruct KSM to only collect or expose specific resources or metric families using command-line arguments. This reduces both KSM memory utilization and network output.
--resources: Restricts KSM to watch specific Kubernetes objects only (e.g., ignore configmaps or secrets).
--metric-allowlist: Explicitly lists metric families to expose (all others are dropped).
--metric-denylist: Drops specific metric families while keeping the rest.
Setting via Helm (values.yaml):
# Helm configuration for prometheus-community/kube-state-metrics
resources:
# Watch only specific K8s API resources
resources:
- pods
- deployments
- statefulsets
- cronjobs
- nodes
# Disable specific metric families across enabled resources
extraArgs:
- --metric-denylist=kube_configmap_info,kube_secret_info,kube_pod_labels
Setting via Argo CD Application Manifest:YAMLapiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: kube-state-metrics
namespace: argocd
spec:
# ... (repo configuration omitted)
source:
helm:
extraArgs:
- --resources=pods,deployments,cronjobs,nodes
- --metric-denylist=kube_pod_created,kube_pod_completion_time
Method 2: Dropping Metrics at Prometheus Ingestion
If you do not manage KSM CLI flags directly, configure Prometheus or Grafana Agent/Alloy to drop metrics during the scrape phase using metric_relabel_configs:
Prometheus CustomResource (Prometheus / ServiceMonitor):
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: kube-state-metrics
namespace: monitoring
spec:
endpoints:
- port: http-metrics
metricRelabelings:
# Drop entire metric families matching a regex pattern
- action: drop
sourceLabels: [__name__]
regex: "(kube_configmap_.*|kube_secret_.*|kube_lease_.*)"
# Drop high-cardinality container state metrics if unused
- action: drop
sourceLabels: [__name__]
regex: "kube_pod_container_state_started"
kube_job_status_failed Source Filtering vs. Scraper DroppingFeatureKSM CLI Flags (--resources, --metric-denylist)Prometheus metric_relabel_configsKSM Memory UsageLower (KSM does not store or process ignored objects in memory)Unchanged (KSM processes objects regardless)Network TrafficLower (HTTP payload size is smaller)Unchanged (KSM transmits full payload; Prometheus discards post-fetch)Prometheus DB StorageLowerLowerConfig PlacementKSM Deployment manifestPrometheus scrape configuration
Prometheus Metrics Exposed by kube-state-metrics
kube_job_status_succeeded
kube_job_status_succeeded is a Prometheus metric exposed by kube-state-metrics that tracks the status of Kubernetes Jobs.
It returns a gauge value representing whether a Job execution succeeded, partitioned by label selectors.
Metric Breakdown
- Metric Name: kube_job_status_succeeded
- Type: Gauge
- Value:
- 1: The Job has completed successfully.
- 0: The Job is in progress, failed, or has not succeeded.
Key Labels
Label Description
==== =========
job_name The name of the Kubernetes Job resource
namespace The namespace where the Job resides
condition Condition status (typically true when checking successful completion)
Common PromQL Queries
1. List All Currently Succeeded Jobs
kube_job_status_succeeded{condition="true"} == 1
2. Detect Jobs That Failed or Did Not Succeed
kube_job_status_succeeded{condition="true"} == 0
3. Alerting Rule: Job Failure
Alert when a Job has completed its execution attempt without succeeding:
- alert: KubernetesJobFailed
expr: kube_job_status_failed{condition="true"} == 1
for: 5m
labels:
severity: warning
annotations:
summary: "Kubernetes Job {{ $labels.namespace }}/{{ $labels.job_name }} failed"
4. Track Job Completion Rate Over Time
Calculate the rate of successful Job completions across a cluster:
sum(increase(kube_job_status_succeeded{condition="true"}[1h]))
kube_job_status_failed
kube_job_status_failed is a Prometheus metric exposed by kube-state-metrics that tracks whether a Kubernetes Job has failed to execute successfully.
It returns a gauge value representing the failure status of a Job, partitioned by label selectors.
Metric Breakdown
- Metric Name: kube_job_status_failed
- Type: Gauge
- Value:
- 1: The Job reached its failure condition (e.g., exceeded backoffLimit or failed execution).
- 0: The Job has not failed (it is currently running, pending, or succeeded).
Key Labels
Label Description
==== =========
job_name The name of the Kubernetes Job resource
namespace The namespace where the Job resides
condition Condition status (typically true when evaluating an active failure state)
reason The reason for failure if populated (e.g., BackoffLimitExceeded, DeadlineExceeded)
Common PromQL Queries
1. List All Currently Failed Jobs
kube_job_status_failed{condition="true"} == 1
2. Count Failed Jobs by Namespace
sum by (namespace) (kube_job_status_failed{condition="true"} == 1)
3. Prometheus Alerting Rule for Job Failures
Alert when a Job has reached a failed condition and remained failed for 5 minutes:
- alert: KubernetesJobFailed
expr: kube_job_status_failed{condition="true"} == 1
for: 5m
labels:
severity: warning
annotations:
summary: "Kubernetes Job {{ $labels.namespace }}/{{ $labels.job_name }} failed"
description: "Job {{ $labels.job_name }} in namespace {{ $labels.namespace }} failed to complete."
4. Filter CronJobs / Generated Jobs by Prefix
If Jobs are generated dynamically by CronJobs, group or filter using regular expressions:
kube_job_status_failed{condition="true", job_name=~"nightly-backup-.*"} == 1
kube_cronjob_status_last_successful_time
kube_cronjob_status_last_successful_time is a gauge metric exposed by kube-state-metrics.
It records the Unix timestamp (in seconds) of when a Kubernetes CronJob last completed execution successfully.
Metric Details
Attribute:Value
- Exporter: kube-state-metrics
- Metric Type:Gauge
- Value: Unix timestamp (seconds) or omitted if the CronJob has never completed
- Labels: cronjob, namespace
Common PromQL Use Cases
1. Time Since Last Successful Run
Calculates how many seconds have elapsed since the CronJob last succeeded:
time() - kube_cronjob_status_last_successful_time
2. Alert on CronJob Failure or Missed Schedule
Alerts if the time since the last successful execution exceeds a target threshold (e.g., 1 day / 86,400 seconds):
(time() - kube_cronjob_status_last_successful_time) > 86400
3. Alerting when Schedule Run Failed
Compares the last scheduled time (kube_cronjob_status_last_schedule_time) against the last successful time. If last_schedule_time is greater than last_successful_time, the most recent run failed or is taking abnormally long:
kube_cronjob_status_last_schedule_time - kube_cronjob_status_last_successful_time > 0
4. Filter out Suspended CronJobs
Combine with kube_cronjob_spec_suspend to suppress false alerts on intentionally paused jobs:
(
(time() - kube_cronjob_status_last_successful_time) > 86400
) and on (cronjob, namespace) (
kube_cronjob_spec_suspend == 0
)
No comments:
Post a Comment