How to set usage limits for ephemeral storage in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Ephemeral Storage

Issue

  • The solution How to set quota for emptyDir volume usage on an Openshift Node? does not work in OpenShift 4. How can I protect my OpenShift 4 Compute Nodes from random writes to temporary container filesystems?

  • How can the amount of ephemeral storage be limited in a Pod?

  • Pods are evicted and the events show messages like:

    Pod ephemeral local storage usage exceeds the total limit of containers

Resolution

If pods are evicted with messages like "Pod ephemeral local storage usage exceeds the total limit of containers", it would be needed to increase the limits of the ephemeral-storage in the Pods or in the ResourceQuotas/LimitRanges.

Configuring the resources.requests.ephemeral-storage and resources.limits.ephemeral-storage for the containers, the sum of ephemeral storage limits across pods in a non-terminal state cannot exceed the value set:

apiVersion: v1
kind: Pod
[...]
spec:
  containers:
  [...]
    resources:
      limits:
        ephemeral-storage: 200Mi
      requests:
        ephemeral-storage: 100Mi
[...]

This configuration can be used in ResourceQuotas as well as LimitRanges to apply project limitation respectively enforce a specific amount if nothing is defined. More details can be found in Resource quotas per project and Restrict resource consumption with limit ranges.

As an example, using LimitRange to enforce the setting of ephemeral-storage can be done using the following definition:

apiVersion: v1
kind: LimitRange
metadata:
  name: storage-limits
spec:
  limits:
    - type: Pod
      max:
        ephemeral-storage: 1Gi
    - type: Container 
      default:
        ephemeral-storage: 100Mi
      defaultRequest:
        ephemeral-storage: 100Mi
      max:
        ephemeral-storage: 500Mi

Root Cause

It's possible to configure limits for the ephemeral storage used by containers and pods. For additional information, refer to How ephemeral storage works in OCP4.

Diagnostic Steps

Check the events for messages about evicted pods when exceeding the amount of storage that is defined:

$ oc get events -n [namespace_name] --sort-by='.lastTimestamp'

[...]
100s        Warning   Evicted             pod/httpd-7579cb6c87-jzwbc          Pod ephemeral local storage usage exceeds the total limit of containers 200Mi.
[...]

Check if there are ResourceQuotas, ClusterResourceQuotas or LimitRanges defined in the cluster:

$ oc get resourcequotas -A
[...]
$ oc get clusterresourcequotas
[...]
$ oc get limitranges -A
[...]

Refer to How to monitor ephemeral storage consumption at pod level? to check the ephemeral storage consumption.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.