etcd_object_counts metric only show OpenShift resources

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• etcd metrics

Issue

• Reviewing the dashboard in console-openshift-console.apps.[example.com]/monitoring/dashboards/grafana-dashboard-api-performance?apiserver=kube-apiserver, it shows only OpenShift related resources in "etcd Object Count".
• etcd_object_counts only reports OpenShift resources and is missing kubernetes core resources.
• Starting with OCP 4.10, etcd_object_counts does no longer correctly report number of objects per resources.

Resolution

Metric etcd_object_counts was deprecated, and apiserver_storage_objects should be used instead when attempting to query resources within etcd.

The issue with the wrong metric used in the dashboard was tracked in This content is not included.bug OCPBUGS-5873 and fixed in OpenShift 4.13.0 by errata RHSA-2023:1326.

Some promql query examples for using the apiserver_storage_objects metric:

• Total number of resources in etcd (any of):

  sum(resource:apiserver_storage_objects:max)
  
  sum(max(apiserver_storage_objects) by (resource))
  

• Show the top 15 resources (with regards to number of objects) in etcd (any of):

  topk(15, resource:apiserver_storage_objects:max)
  
  topk(15, max(apiserver_storage_objects) by (resource))
  

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Metric etcd_object_counts was Content from github.com is not included.replaced in Kubernetes 1.21, in which OpenShift 4.8 is based, and apiserver_storage_objectswas promoted and marked as stable in Content from github.com is not included.Kubernetes 1.21.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.