Monitor ephemeral storage consumption by individual pods in RHOCP

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Ephemeral storage

Issue

• How to monitor ephemeral storage consumption at pod level?
• Is there a Prometheus metric that shows the ephemeral storage consumption?
• Does the metric container_fs_usage_bytes, that is used for the storage graph shown on Pod's details tab, take the ephemeral storage into account?

Resolution

There is currently no exposed metric for ephemeral storage consumed by a Pod.

To get the current ephemeral storage usage, it is possible to check the output of below command to verify how much ephemeral-storage every Pod is using:

$ oc get nodes -o custom-columns=":.metadata.name" --no-headers | while read NODE; do  printf "\n===> $NODE\n" && oc get --raw /api/v1/nodes/${NODE}/proxy/stats/summary  | jq '.pods[] | . as $parent | $parent.podRef.namespace + " " + $parent.podRef.name + " " + ($parent."ephemeral-storage".usedBytes/1024|tostring)' -r | sort -k3 -hr | column -t;  done

It is possible to filter by specific nodes, like for example, for worker nodes, adding -l node-role.kubernetes.io/worker= to the oc get nodes command above.

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Kubernetes does not expose metrics on emptyDir as it does for persistent volumes. The metric container_fs_usage_bytes shows the usage of the filesystem within the Container, but emptyDir is a Volume created at the Pod level and is a shared resource for all Containers in the Pod, thus it is not counted with this metric. As there is currently no exposed metric for ephemeral storage used by Pods, there is an upstream request for the same: Content from github.com is not included.exposing ephemeral storage metrics to prometheus.

Refer to Content from kubernetes.io is not included.ephemeral storage consumption management for information about how the kubelet measures storage use, and when pods are marked for eviction due to ephemeral storage consumption.Refer also to Content from kubernetes.io is not included.emptyDir for additional information.

Important notes to take in consideration when using ephemeral storage as per Content from kubernetes.io is not included.configurations for local ephemeral storage:

• The kubelet tracks tmpfs emptyDir volumes as container memory use, rather than as local ephemeral storage.
• The kubelet will only track the root filesystem for ephemeral storage. OS layouts that mount a separate disk to /var/lib/kubelet or /var/lib/containers will not report ephemeral storage correctly.

Diagnostic Steps

It is possible to check specific Pod running in specific Node as per the following example:

$ oc get --raw /api/v1/nodes/[node_name]/proxy/stats/summary | jq -r '.pods[] | select(.podRef.name == "[pod_name]") | ."ephemeral-storage"'
{
  "time": "2023-07-10T16:40:31Z",
  "availableBytes": 26224336896,
  "capacityBytes": 42344624128,
  "usedBytes": 3491913728,
  "inodesFree": 20348799,
  "inodes": 20708800,
  "inodesUsed": 259
}

$ oc get --raw /api/v1/nodes/[node_name]/proxy/stats/summary | jq -r '.pods[] | select(.podRef.name == "[pod_name]") | ."ephemeral-storage".usedBytes' | xargs numfmt --to iec
3.3G

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.