Resolving "Detected modified SecurityContextConstraints" update gate before upgrading to 4.14

Environment

OCP 4.13

Issue

An update to 4.14 cannot be triggered on a 4.13 cluster, showing the following message:

Upgradeable=False

  Reason: DetectedModifiedSccResources
  Message: Detected modified SecurityContextConstraints: <name1>, <name2>. These modifications would be removed by the update. Please ensure all cluster workloads are able to run without these usupported modifications, then delete the modified SCC resources (they will be recreated without the modifications). See https://access.redhat.com/solutions/7033949 for more information.

Resolution

For each SCC listed in the message as detected, do the following. This article uses restricted-v2 as an example SCC but your cluster may show different ones as modified.

1. Backup the SCC resources listed in the message

$ oc get scc restricted-v2 -o yaml > scc-restricted-v2.yaml

2. Delete the SCC resources listed in the message

$ oc delete scc restricted-v2 -o yaml

3. Once all modified SCCs were deleted and recreated this way, the Upgradeable=False gate disappears (it can take several minutes). While this allows you to initiate the update to 4.14 now, make sure you validate the health of your workloads first, following the next step.
4. Verify all workloads in the clusters are able to run. The SCCs affect pods on startup, so monitor the cluster for Pods stuck in the Pending state. You may consider forcing a rollout of all Deployments to discover whether Pods are able to start up.
   • You may need to adjust specification of problematic Pods (or a higher-level resource such as Deployment) towards what the workload actually requires
   • Alternatively, you may need to create new custom SCCs resources to permit the Pods to execute
   • Consult documentation on This page is not included, but the link has been rewritten to point to the nearest parent document.Managing security context constraints and Configuring seccomp profiles
5. Once you verified that your workloads are safe, initiate the upgrade with oc adm upgrade or via the Web Console.

Root Cause

Modifying system SecurityContextConstraint (SCC) resources is not supported and OCP can revert user modifications at any time. Because of a defect, OCP versions before 4.14 were not doing so. This was fixed in OCP 4.14 and later (see This content is not included.OCPBUGS-18386). Because existing workloads can depend on earlier modifications, in OCP versions 4.13 and lower, this defect was fixed in a way that tolerates user modifications (see This content is not included.OCPBUGS-19472), and an update gate (Upgradeable=False status condition) was set up between OCP versions 4.13 and 4.14 to prevent breaking workloads.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.