OpenShift reports SELinux-related conflicts when creating Pods

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.20 and above

Issue

SELinux related events observed when starting a Pod:

`SELinuxLabel ‘XXX’ conflicts with pod YYY that uses the same volume as this pod with SELinuxLabel ‘ZZZ’. If both pods land on the same node, only one of them may access the volume.`

`SELinuxLabel ‘XXX’ conflicts with another pod that uses the same volume as this pod with a different SELinuxLabel. If both pods land on the same node, only one of them may access the volume.`

`SELinuxChangePolicy ‘XXX’ conflicts with another pod that uses the same volume as this pod with a different SELinuxChangePolicy. If both pods land on the same node, only one of them may access the volume.`

Resolution

Each event must be validated separately:

  • Event: SELinuxLabel ‘XXX’ conflicts with pod YYY that uses the same volume as this pod with SELinuxLabel ‘ZZZ’.

    Considerations:

    • Understand the nature of the conflict. This event indicates that the Pod associated with this event is using the same volume as Pod YYY, and they have different SELinux labels (XXX and ZZZ). This may be permissible currently, but in a future OCP release, when the SELinuxMount feature gate graduates to General Available (GA), all Pods concurrently accessing the same volume will need to have identical SELinux labels. At this point, nothing is broken, and all Pods exhibiting these events will start. This is merely preparation for a breaking change. The future OCP release will prevent cluster upgrades until all such conflicts are resolved, ensuring that an upgrade does not break existing applications.

    • Check SELinux labels used by the Pods reported in the event. These can be set either at the Pod level, in spec.securityContext.seLinuxOptions, or within the individual containers, in spec.containers[*].securityContext.seLinuxOptions.

      • An empty seLinuxOptions in unprivileged Pods means that the container runtime (CRI-O) will assign a random SELinux label to the Pod upon startup. CRI-O will then always recursively relabel all files on the Pod volumes to this unique value. Consequently, no other Pod can access the Pod’s volumes, except for privileged Pods.

      • Privileged containers always start with spc_t SELinux label, regardless of their SELinux label specified in Pod’s spec. Privileged containers can access any file on the host or in any other container, especially with any SELinux label.

    You have 2 options to resolve this issue:

    • Option 1: Set SELinux labels on all these Pods to the same value. Typically, by editing the owners of the Pods, such as StatefulSets or Deployments. When all Pods use the same SELinux label and the default SELinuxChangePolicy, OCP will mount the volume in constant time with the -o context=XXX mount option in the future OCP release and thus the Pods will start faster.

    • Option 2: Set SELinuxChangePolicy to Recursive in all conflicting Pods. This is useful when it’s impossible or undesirable to set the same SELinux labels on these Pods. For example, when a privileged Pod (running with spc_t label) and unprivileged Pod (running with some cXX,cYY:s0 label) want to share data on a volume.

      • Using seLinuxChangePolicy: Recursive opts-out the Pod from SELinuxMount feature and the container runtime will keep relabeling the Pod volumes recursively. It also disables the event + related metrics, even when multiple Pods use the same volume with different SELinux labels.

      • With the privileged + unprivileged Pod example above, the privileged pod starts with label spc_t and the container runtime won’t relabel any files on the volume. When the un-privileged one starts, the container runtime will relabel all files to the Pod’s SELinux label. As a result, both Pods can access the files on the volume. But both must set their seLinuxChangePolicy to Recursive, because the mount option can’t be used.

      • You can set the default seLinuxChangePolicy at the namespace level by setting the namespace label storage.openshift.io/selinux-change-policy to Recursive. Note that it will affect only newly started Pods in the namespace:

        $ oc label namespace <namespace name> storage.openshift.io/selinux-change-policy=Recursive

As a concrete example, consider these two Deployments, each with a single replica:

  1. Deployment myapp, using volume myclaim, running with restricted SCC. Due to the restricted SCC, the Pods of this Deployment inherit their SELinux label from its namespace, say level: s0:c27,c19:
$ oc get pod myapp-545f9bb5d9-hswnp --o yaml
...
spec:
  securityContext:
    seLinuxOptions:
      level: s0:c27,c19
...
  volumes:
  - name: vol
    persistentVolumeClaim:
      claimName: myclaim
  1. Deployment my-privileged-app, running a privileged container, with no explicit SELinux label and using the same PVC:
$ oc get pod my-privileged-app-7c79944f6-x69r9 --o yaml
spec:
  containers:
  - name: scanner
    securityContext:
      privileged: true
...

  volumes:
  - name: vol
    persistentVolumeClaim:
      claimName: myclaim

While both Pods can run on OCP 4.20 and access the volume, it will not be possible when the SELinuxMount feature gate is promoted to GA. To warn users about it, OCP sends events to both of them when it detects such a conflict:

$ oc describe pod myapp-545f9bb5d9-hswnp
...
Events:
Type Reason                 Age From            Message
---- ------                 --- ----            --------
Normal SELinuxLabelConflict 45s selinux_warning SELinuxLabel ":::s0:c27,c19" conflicts with pod my-privileged-app-7c79944f6-x69r9 that uses the same volume as this pod with SELinuxLabel "". If both pods land on the same node, only one of them may access the volume.

$ oc describe pod my-privileged-app-7c79944f6-x69r9
...
Events:
Type Reason                 Age From            Message
---- ------                 --- ----            --------
Normal SELinuxLabelConflict 45s selinux_warning SELinuxLabel "" conflicts with pod myapp-545f9bb5d9-hswnp that uses the same volume as this pod with SELinuxLabel ":::s0:c27,c19". If both pods land on the same node, only one of them may access the volume.

Since the user needs both Pods as they are, one unprivileged, with a SELinux label inherited from the SCC and one privileged, the user can decide to set seLinuxChangePolicy in both Deployments to ensure the volume is always relabelled recursively and to silence the event:

$ oc patch deployment myapp -p '{"spec": {"template": {"spec": {"securityContext": {"seLinuxChangePolicy": "Recursive"}}}}}'
$ oc patch deployment my-privileged-app -p '{"spec": {"template": {"spec": {"securityContext": {"seLinuxChangePolicy": "Recursive"}}}}}'

Alternatively, the user can choose to default to Recursive policy in the whole namespace. This will affect only newly started Pods:

$ oc label namespace myapp "storage.openshift.io/selinux-change-policy=Recursive"

  • Event: “SELinuxLabel ‘XXX’ conflicts with another pod that uses the same volume as this pod with a different SELinuxLabel.”

    This is just a variant of the previous event when the conflicting Pods are in different namespaces. OCP won’t reveal the name of the Pods, as it would allow unprivileged users to peek into other namespace Pod names. To find the exact names of the conflicting Pods, use the metric described below in Diagnostic steps. Only users with elevated privileges, such as the cluster administrator, can see the metrics.

  • Event: “SELinuxChangePolicy ‘XXX’ conflicts with another pod that uses the same volume as this pod with a different SELinuxChangePolicy.”

    All Pods that concurrently access the same volume need to have the same seLinuxChangePolicy.

    • Check the spec.securityContext.seLinuxChangePolicy of all Pods reported in the event and make sure it’s the same.

    • You can set the default seLinuxChangePolicy at the namespace level, by setting the namespace label storage.openshift.io/selinux-change-policy to Recursive. Note that it will affect only newly started Pods in the namespace.

Root Cause

Currently, the Pod SELinux label may be applied using the mount option -o context=<pod_selinux_label> for PersistentVolumes with spec.accessMode: [ReadWriteOncePod]. This applies the SELinux label to Pod volume in a constant time, regardless of the amount of files on the volume.
For all other PVs accessMode, the SELinux label is applied recursively to all files on the volume, which can take a significant amount of time for volumes with many files.
We plan to apply the SELinux label using the mount option to all PersistentVolumes in a future OCP release. Since a volume can be mounted with only a single such option, all Pods that concurrently access the volume will have to use the same SELinux label. In addition, the label must be known when starting the first Pod that uses the volume. OCP expects that such a change will be fine for the vast majority of applications.` OCP reports any possible breakage with the events described above and offers cluster or application administrators to opt out from the new behavior and use the same recursive relabeling as in the previous OCP releases.

Diagnostic Steps

  1. Check Pod Events: Look for SELinux-related events on the Pods. For example using oc describe pod <pod name> or oc get event.
  2. Check Cluster Metrics: Examine the selinux_warning_controller_selinux_volume_conflict metric in the cluster. This metric reports conflicting Pods and their namespaces, revealing property values (SELinux label or change policy).
propertypod1_namepod1_namespacepod1_valuepod2_namepod2_namespacepod2_valueValue
SELinuxLabelmy-privileged-app-7c79944f6-x69r9myappmyapp-545f9bb5d9-hswnpmyapp:::s0:c27,c191
SELinuxLabelmyapp-545f9bb5d9-hswnpmyapp:::s0:c27,c19my-privileged-app-7c79944f6-x69r9myapp1

The property label has value SELinuxLabel, if the pods have conflicting SELinux label, or SELinuxChangePolicy, if the pods have conflicting spec.securityContext.SELinuxChangePolicy. pod1_value and pod2_value then show the property value - either SELinux label or the change policy. Each conflict is reported twice, with both combinations of pod1 and pod2 to make filtering per pod or per namespace easier.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.