Understanding OpenShift File System Monitoring (eviction conditions)

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

If you want to understand how OpenShift monitors your host filesystems, to ensure that you are alerted to conditions that could affect the stability of your OpenShift cluster, it's important to understand what OpenShift is monitoring and how it is monitoring those filesystems.

To best understand what OpenShift monitors we can refer to the Content from kubernetes.io is not included.upstream kubernetes documentation

`kubelet` supports only two filesystem partitions.

The `nodefs` filesystem that `kubelet` uses for volumes, daemon logs, etc.
The `imagefs` filesystem that container runtimes uses for storing images and container writable layers.

`imagefs` is optional. `kubelet` auto-discovers these filesystems using cAdvisor. `kubelet` does not care about any other filesystems. Any other types of configurations are not currently supported by the `kubelet`. For example, it is _not OK_ to store volumes and logs in a dedicated filesystem.

• Note: for both the nodefs and imagefs due to how RHCOS or the RHEL standard filesystem is laid out on disk, it's likely that the kubelet will only monitor the nodefs filesystem, which is mapped to /. Explicit steps have to be taken with a host's storage layout to change this behavior.
  • It is for this reason, that Red Hat only recommends making host-level storage changes that align with the monitoring capabilities of the platform. For example, you should only supply additional disks, in a raid or lvm format such that a single filesystem is presented to the imagefs, leaving the nodefs on the '/' filesystem.

OpenShift monitors the following metrics and defines the following eviction thresholds for garbage collection. Please refer to the product docmentation to alter any eviction values.

nodefs.available

: From cadvisor, the node.stats.fs.availablemetric represents how much fs space is available (remaining) on the nodes filesystem (where /var/lib/kubelet resides).
Default Eviction Threshold: <10%

nodefs.inodesFree

: From cadvisor, the node.stats.fs.inodesFree metric represents the number of free inodes left in the node's filesystem. This filesystem is determined based on where /var/lib/kubelet resides, with RHCOS or a standard RHEL FS layout this is the filesystem mapped to /.
Default Eviction Threshold: <5%

imagefs.available

: From cadvisor, the node.stats.runtime.imagefs.availablemetric represents how much fs space is available (remaining) on the image storage filesystem (where /var/lib/containers/ resides).
Default Eviction Threshold: <15%

imagefs.inodesFree

: From cadvisor, the node.stats.runtime.imagefs.inodesFree metric represents the number of free inodes left in the image storage filesystem.

• Note: using df -h <filesystem mount point> and df -i <filesystem mount point> where is either the nodefs or the imagefs, should show the same metrics pulled by cadvisor.

Readers of this article should also refer to our resource reserve documentation, for more information on improving the reliability/stability of nodes.