Skip to main content

Kubernetes Cluster State

Kubernetes Cluster State

Plugin: go.d.plugin Module: k8s_state

Maintained by Netdata

Overview​

This collector monitors Kubernetes Nodes, Pods, Containers, Deployments, CronJobs and PersistentVolumeClaims.

This collector is supported on all platforms.

This collector only supports collecting metrics from a single instance of this integration.

The collector reads cluster state from the Kubernetes API with the permissions below, granted cluster-wide (all namespaces). It only reads cluster objects; the access review is a query that stores nothing.

ResourceAPI groupVerbsUsed for
nodescore ("")list, watchNode metrics.
podscore ("")list, watchPod and container metrics, and the pod and container counts of each Node.
deploymentsappslist, watchDeployment metrics.
cronjobsbatchlist, watchCronJob metrics.
jobsbatchlist, watchThe Job runs behind the CronJob metrics (status, failure reason, duration).
persistentvolumeclaimscore ("")list, watchPVC metrics. Optional, see below.
namespacescore ("")getThe UID of the kube-system Namespace, used as the k8s_cluster_id label. Without it the label is empty and a warning is logged.
selfsubjectaccessreviewsauthorization.k8s.iocreateA check at startup that the persistentvolumeclaims permissions are granted. The default system:basic-user ClusterRole gives it to every authenticated account.
/version (non-resource URL)n/agetThe connection check when the job starts. The default system:public-info-viewer ClusterRole gives it to every account.

The official Netdata Helm chart grants every row down to namespaces (persistentvolumeclaims from the release that adds PVC metrics). With your own ClusterRole, add those rows; the last two are allowed by the default Kubernetes ClusterRoles.

Without the persistentvolumeclaims permissions the PVC metrics are skipped with one warning and all other metrics work. They are checked once, at startup, so permissions granted later take effect after the collector is restarted.

Kubernetes Cluster State can be monitored further using the following other integrations:

Default Behavior​

Auto-Detection​

This integration doesn't support auto-detection.

Performance Impact​

The default configuration for this integration is not expected to impose a significant performance impact on the system.

Setup​

You can configure the k8s_state collector in two ways:

MethodBest forHow to
UIFast setup without editing filesGo to Nodes → Configure this node → Collectors → Jobs, search for k8s_state, then click + to add a job.
FileIf you prefer configuring via file, or need to automate deployments (e.g., with Ansible)Edit go.d/k8s_state.conf and add a job.
important

UI configuration requires paid Netdata Cloud plan.

Prerequisites​

No action required.

Configuration​

Options​

There are no configuration options.

via UI​

Configure the k8s_state collector from the Netdata web interface:

  1. Go to Nodes.
  2. Select the node where you want the k8s_state data-collection job to run and click the ⚙ (Configure this node). That node will run the data collection.
  3. The Collectors → Jobs view opens by default.
  4. In the Search box, type k8s_state (or scroll the list) to locate the k8s_state collector.
  5. Click the + next to the k8s_state collector to add a new job.
  6. Fill in the job fields, then click Test to verify the configuration and Submit to save.
    • Test validates the provided settings and checks the collector's startup prerequisites. Successful validation does not guarantee that every metric will be available during collection.
    • If it fails, an error message appears with details (for example, connection refused, timeout, or command execution errors), so you can adjust and retest.

via File​

The configuration file name for this integration is go.d/k8s_state.conf.

The file format is YAML. Generally, the structure is:

update_every: 1
autodetection_retry: 0
jobs:
- name: some_name1
- name: some_name2

You can edit the configuration file using the edit-config script from the Netdata config directory.

cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config go.d/k8s_state.conf
Examples​

There are no configuration examples.

Alerts​

The following alerts are available:

Alert nameOn metricDescription
k8s_state_deployment_condition_available k8s_state.deployment_conditionsDeployment ${label:k8s_deployment_name} does not have the minimum required replicas
k8s_state_cronjob_last_execution_failed k8s_state.cronjob_last_execution_statusCronJob ${label:k8s_cronjob_name} in ${label:k8s_namespace} failing

Metrics​

Metrics grouped by scope.

The scope defines the instance that the metric belongs to. An instance is uniquely identified by a set of labels.

Per node​

These metrics refer to the Node.

Labels:

LabelDescription
k8s_cluster_idCluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_nameCluster name. Cluster name discovery only works in GKE.
k8s_node_nameNode name.

Metrics:

MetricDescriptionDimensionsUnit
k8s_state.node_allocatable_cpu_requests_utilizationCPU requests utilizationrequests%
k8s_state.node_allocatable_cpu_requests_usedCPU requests usedrequestsmillicpu
k8s_state.node_allocatable_cpu_limits_utilizationCPU limits utilizationlimits%
k8s_state.node_allocatable_cpu_limits_usedCPU limits usedlimitsmillicpu
k8s_state.node_allocatable_mem_requests_utilizationMemory requests utilizationrequests%
k8s_state.node_allocatable_mem_requests_usedMemory requests usedrequestsbytes
k8s_state.node_allocatable_mem_limits_utilizationMemory limits utilizationlimits%
k8s_state.node_allocatable_mem_limits_usedMemory limits usedlimitsbytes
k8s_state.node_allocatable_pods_utilizationPods resource utilizationallocated%
k8s_state.node_allocatable_pods_usagePods resource usageavailable, allocatedpods
k8s_state.node_conditionCondition statusReady, DiskPressure, MemoryPressure, NetworkUnavailable, PIDPressurestatus
k8s_state.node_schedulabilitySchedulabilityschedulable, unschedulablestate
k8s_state.node_pods_readinessPods readinessready%
k8s_state.node_pods_readiness_statePods readiness stateready, unreadypods
k8s_state.node_pods_conditionPods conditionpod_ready, pod_scheduled, pod_initialized, containers_readypods
k8s_state.node_pods_phasePods phaserunning, failed, succeeded, pendingpods
k8s_state.node_containersContainerscontainers, init_containerscontainers
k8s_state.node_containers_stateContainers staterunning, waiting, terminatedcontainers
k8s_state.node_init_containers_stateInit containers staterunning, waiting, terminatedcontainers
k8s_state.node_ageAgeageseconds

Per deployment​

These metrics refer to Deployments.

Labels:

LabelDescription
k8s_cluster_idCluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_nameCluster name. Cluster name discovery only works in GKE.
k8s_deployment_nameDeployment name.
k8s_namespaceNamespace.

Metrics:

MetricDescriptionDimensionsUnit
k8s_state.deployment_conditionsDeployment Conditionsavailable, replica_failure, progressingstatus
k8s_state.deployment_replicasDeployment Replicasdesired, current, readyreplicas
k8s_state.deployment_ageDeployment Ageageseconds

Per cronjob​

These metrics refer to CronJobs.

Labels:

LabelDescription
k8s_cluster_idCluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_nameCluster name. Cluster name discovery only works in GKE.
k8s_cronjob_nameCronJob name.
k8s_namespaceNamespace.

Metrics:

MetricDescriptionDimensionsUnit
k8s_state.cronjob_jobs_count_by_statusCronJob Jobs Count by Statuscompleted, failed, running, suspendedjobs
k8s_state.cronjob_jobs_failed_by_reasonCronJob Jobs Failed by Reasonpod_failure_policy, backoff_limit_exceeded, deadline_exceededjobs
k8s_state.cronjob_last_execution_statusCronJob Last Execution Statuscompleted, failedstatus
k8s_state.cronjob_last_completion_durationCronJob Last Completion Durationlast_completionseconds
k8s_state.cronjob_last_completed_time_agoCronJob Last Completed Time Agolast_completed_agoseconds
k8s_state.cronjob_last_schedule_time_agoCronJob Last Schedule Time Agolast_schedule_agoseconds
k8s_state.cronjob_suspend_statusCronJob Suspend Statusenabled, suspendedstatus
k8s_state.cronjob_ageCronJob Ageageseconds

Per pod​

These metrics refer to the Pod.

Labels:

LabelDescription
k8s_cluster_idCluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_nameCluster name. Cluster name discovery only works in GKE.
k8s_node_nameNode name.
k8s_namespaceNamespace.
k8s_controller_kindController kind (ReplicaSet, DaemonSet, StatefulSet, Job, etc.).
k8s_controller_nameController name.
k8s_pod_namePod name.
k8s_qos_classPod QOS class (burstable, guaranteed, besteffort).

Metrics:

MetricDescriptionDimensionsUnit
k8s_state.pod_cpu_requests_usedCPU requests usedrequestsmillicpu
k8s_state.pod_cpu_limits_usedCPU limits usedlimitsmillicpu
k8s_state.pod_mem_requests_usedMemory requests usedrequestsbytes
k8s_state.pod_mem_limits_usedMemory limits usedlimitsbytes
k8s_state.pod_conditionConditionpod_ready, pod_scheduled, pod_initialized, containers_readystate
k8s_state.pod_phasePhaserunning, failed, succeeded, pendingstate
k8s_state.pod_status_reasonStatus reasonEvicted, NodeAffinity, NodeLost, Shutdown, UnexpectedAdmissionError, Otherstatus
k8s_state.pod_ageAgeageseconds
k8s_state.pod_containersContainerscontainers, init_containerscontainers
k8s_state.pod_containers_stateContainers staterunning, waiting, terminatedcontainers
k8s_state.pod_init_containers_stateInit containers staterunning, waiting, terminatedcontainers

Per container​

These metrics refer to the Pod container.

Labels:

LabelDescription
k8s_cluster_idCluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_nameCluster name. Cluster name discovery only works in GKE.
k8s_node_nameNode name.
k8s_namespaceNamespace.
k8s_controller_kindController kind (ReplicaSet, DaemonSet, StatefulSet, Job, etc.).
k8s_controller_nameController name.
k8s_pod_namePod name.
k8s_qos_classPod QOS class (burstable, guaranteed, besteffort).
k8s_container_nameContainer name.

Metrics:

MetricDescriptionDimensionsUnit
k8s_state.pod_container_readiness_stateReadiness statereadystate
k8s_state.pod_container_restartsRestartsrestartsrestarts
k8s_state.pod_container_stateContainer staterunning, waiting, terminatedstate
k8s_state.pod_container_waiting_state_reasonContainer waiting state reasonContainerCreating, CrashLoopBackOff, CreateContainerConfigError, CreateContainerError, ErrImagePull, ImagePullBackOff, InvalidImageName, PodInitializing, Otherstate
k8s_state.pod_container_terminated_state_reasonContainer terminated state reasonCompleted, ContainerCannotRun, DeadlineExceeded, Error, Evicted, OOMKilled, Otherstate

Per persistent volume claim​

These metrics refer to the PersistentVolumeClaim.

  • Requested Storage shows the storage requested by the PVC. Capacity shows the capacity reported in its status. These are Kubernetes allocation values, not filesystem usage or free space.
  • Capacity Shortfall is max(requested - capacity, 0). It is positive while reported capacity is below the request and zero when capacity meets or exceeds the request. A persistent shortfall can indicate an expansion that is pending or failed; it does not identify the cause.
  • Each chart has one dimension. Summing across PVCs gives total requested storage, total reported capacity, or total outstanding requested capacity. Excess capacity on one PVC does not cancel another PVC's shortfall.
  • Unknown values are gaps, not zero. For example, a Pending PVC without reported capacity has no Capacity or Capacity Shortfall values.

Labels:

LabelDescription
k8s_cluster_idCluster ID. This is equal to the kube-system namespace UID.
k8s_cluster_nameCluster name. Cluster name discovery only works in GKE.
k8s_namespaceNamespace.
k8s_pvc_namePersistentVolumeClaim name.

Metrics:

MetricDescriptionDimensionsUnit
k8s_state.pvc_requested_storagePVC Requested StoragerequestedB
k8s_state.pvc_capacityPVC CapacitycapacityB
k8s_state.pvc_capacity_shortfallPVC Capacity ShortfallshortfallB

Troubleshooting​

Diagnostics​

Debug Mode​

Important: Debug mode is not supported for data collection jobs created via the UI using the Dyncfg feature.

To troubleshoot issues with the k8s_state collector, run the go.d.plugin with the debug option enabled. The output should give you clues as to why the collector isn't working.

  • Navigate to the plugins.d directory, usually at /usr/libexec/netdata/plugins.d/. If that's not the case on your system, open netdata.conf and look for the plugins setting under [directories].

    cd /usr/libexec/netdata/plugins.d/
  • Switch to the netdata user.

    sudo -u netdata -s
  • Run the go.d.plugin to debug the collector:

    ./go.d.plugin -d -m k8s_state

    To debug a specific job:

    ./go.d.plugin -d -m k8s_state -j jobName

Getting Logs​

If you're encountering problems with the k8s_state collector, follow these steps to retrieve logs and identify potential issues:

  • Run the command specific to your system (systemd, non-systemd, or Docker container).
  • Examine the output for any warnings or error messages that might indicate issues. These messages should provide clues about the root cause of the problem.
System with systemd​

Use the following command to view logs generated since the last Netdata service restart:

journalctl _SYSTEMD_INVOCATION_ID="$(systemctl show --value --property=InvocationID netdata)" --namespace=netdata --grep k8s_state
System without systemd​

Locate the collector log file, typically at /var/log/netdata/collector.log, and use grep to filter for collector's name:

grep k8s_state /var/log/netdata/collector.log

Note: This method shows logs from all restarts. Focus on the latest entries for troubleshooting current issues.

Docker Container​

If your Netdata runs in a Docker container named "netdata" (replace if different), use this command:

docker logs netdata 2>&1 | grep k8s_state

Do you have any feedback for this page? If so, you can open a new issue on our netdata/learn repository.