How to avoid long SELinux relabelling with ReadWriteOncePod (RWOP) access mode and seLinuxMount in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • SELinux
  • ReadWriteOncePod (RWOP)

Issue

  • Pods take a very long time to start when the volume contains a large number of files.
  • How to avoid SELinux labelling issues while keeping SELinux confining.

Resolution

Starting with OpenShift 4.16, RWOP with SELinux context mount is generally available.

The following resolution uses the CSI Driver seLinuxMount feature, which is limited to volumes with ReadWriteOncePod access mode.

Note: since it is not possible to change access mode of an existing volume, the resolution involves moving data between volumes.

Use the SELinux context mount feature with RWOP

  • Verify that the CSI Driver which is used to provision the PersistentVolume supports the seLinuxMount option. The CSI Driver should have the option seLinuxMount: true:

    # example with seLinuxMount available in the CSIDriver
$ oc get csidriver ebs.csi.aws.com -o yaml | grep seLinuxMount
seLinuxMount: true

    # example with seLinuxMount not available in the CSIDriver
$ oc get csidriver topolvm.io -o yaml | grep seLinuxMount
seLinuxMount: false

  • Identify a PersistentVolumeClaim (PVC) to change the AccessMode from ReadWriteOnce to ReadWriteOncePod to speed up SELinux relabeling, and assess if the workload is compatible with the ReadWriteOncePod mode. The main difference between both, as explained in the OpenShift documentation on Access Modes is:

    • With ReadWriteOnce (RWO), multiple pods on a single node can access the volume.
    • With ReadWriteOncePod (RWOP), a single pod on a single node can access the volume.

  • Following how to migrate data between PVs in OpenShift 4, section Manually migrating data between PVs, create a new PVC with the access mode of the PVC changed from ReadWriteOnce to ReadWriteOncePod.

  • Verify the access mode of the PVC:

    $ oc get pvc my-pvc -o jsonpath='{.spec.accessModes}{"\n"}'
["ReadWriteOncePod"]

  • If using Dynamic Provisioning, a new PV should have been automatically created, with the correct AcccessMode. Verify the AccessMode of the created PV:

    $ oc get pv my-pv -o jsonpath='{.spec.accessModes}{"\n"}'
["ReadWriteOncePod"]

  • Once the AccessMode is changed, the SELinux context of the files contained in the volume will be set at once using the mount(8) context option, instead of recursively changing the extended attributes of each and every file in the volume.

  • Refer also to avoiding long SELinux relabeling times by using the SELinux Mount (Developer Preview) Feature in 4.17 for using the seLinuxMount option with volumes with the RWO or RWX access mode.

Root Cause

A Volume with the ReadWriteOnce access mode can be mounted by multiple pods on the same node. However a SELinux context cannot be different for multiple mounts, hence the need for the ReadWriteOncePod access mode which make sure that the volume is accessed by a single mount on the node.

The ReadWriteOncePod access mode was introduced as a Technology Preview feature in OpenShift 4.14, and RWOP with SELinux context mount is generally available starting with OpenShift 4.16.

Diagnostic Steps

Verify if the seLinuxMount option in combination with ReadWriteOncePod is correctly working by following these steps:

  • Get the SELinux Multi-Category Security (MCS) for the Project in which the Pod is running:

    $ oc get project hello -o jsonpath='{.metadata.annotations.openshift\.io/sa\.scc\.mcs}{"\n"}'
s0:c28,c7

  • Get the Pod UID 

    $ oc get pod <my pod>  -o jsonpath='{.metadata.uid}{"\n"}'
7777df35-e673-45f8-a3c7-32e6052805a9

  • Access the node where the pod is running and inspect the volume mounts of the pod: 

    $ oc debug node/<node>
# chroot /host
# grep /var/lib/kubelet/pods/7777df35-e673-45f8-a3c7-32e6052805a9 /proc/mounts 
tmpfs /var/lib/kubelet/pods/7777df35-e673-45f8-a3c7-32e6052805a9/volumes/kubernetes.io~projected/kube-api-access-bfst6 tmpfs rw,seclabel,relatime,size=14750892k,inode64 0 0
/dev/nvme2n1 /var/lib/kubelet/pods/7777df35-e673-45f8-a3c7-32e6052805a9/volumes/kubernetes.io~csi/pvc-2cd2d838-65b7-4d31-9383-1871d2b73f75/mount ext4 rw,context="system_u:object_r:container_file_t:s0:c7,c28",relatime 0 0

As seen above the SELinux context, including the SELinux MCS s0:c7,c28 is passed as a mount option.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.