Inconsistency of node-role between newly created vs. long running OpenShift 4 clusters

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4

Issue

• It was found that 4.12 and newly created OpenShift clusters have node-role.kubernetes.io/control-plane label set while 4.11 and older OpenShift cluster only apply node-role.kubernetes.io/master label, even after upgrading to 4.12 or newer. Why this is and how to achieve consistency?
• After the upgrade to OCP 4.12, the control-plane role label on master nodes does not get automatically added.
• While troubleshooting an issue it was found that node-role.kubernetes.io/control-plane was applied on OpenShift installed with version 4.12 while it was missing on a cluster installed with a previous version. Even though both are now running OpenShift 4.12 it's confusing to see such differences and this makes troubleshooting hard.
• Why is node-role.kubernetes.io/control-plane not applied to long running OpenShift clusters after upgrading to 4.12 or newer releases?

Resolution

Starting with OpenShift 4.12, all new Control Plane node(s) have the node-role.kubernetes.io/master label and also a new node-role.kubernetes.io/control-plane one set to comply with Kubernetes, and to avoid breaking existing implementation. But clusters installed with a version below 4.12 and upgraded to 4.12 or newer only have the node-role.kubernetes.io/master label applied to the Control Plane node(s).

Starting with OpenShift 4.20 (and that will be backported to some previous versions), the labels for the Control Plane nodes will be automatically updated to include the node-role.kubernetes.io/control-plane label:

Workaround

To achieve consistency between long running OpenShift clusters and newly created ones, it is possible to add the node-role.kubernetes.io/control-plane label to the existing Control Plane node(s) as explained in Understanding how to update labels on nodes with the following command:

$ oc label node -l node-role.kubernetes.io/master node-role.kubernetes.io/control-plane=

Root Cause

As per master role renamed as control-plane in OpenShift 4.12, the node-role.kubernetes.io/control-plane label was introduced to eventually replace the node-role.kubernetes.io/master label at some point to align with efforts such as This content is not included.an update on Red Hat's conscious language efforts.

Diagnostic Steps

• The roles of the Control Plane nodes in a cluster installed with OpenShift 4.9:

    $ oc get nodes
    NAME                                         STATUS   ROLES    AGE     VERSION
    ip-127-0-151-14.cloud.compute.intra          Ready    worker   4h42m   v1.24.15+a9da4a8
    ip-127-0-159-95.cloud.compute.intra          Ready    master   4h48m   v1.24.15+a9da4a8
    ip-127-0-176-215.cloud.compute.intra         Ready    master   4h48m   v1.24.15+a9da4a8
    ip-127-0-190-238.cloud.compute.intra         Ready    worker   4h38m   v1.24.15+a9da4a8
    ip-127-0-197-202.cloud.compute.intra         Ready    worker   4h42m   v1.24.15+a9da4a8
    ip-127-0-207-151.cloud.compute.intra         Ready    master   4h48m   v1.24.15+a9da4a8
  

• The roles of the Control Plane nodes in a cluster installed with OpenShift 4.12 or newer:

    $ oc get nodes
    NAME                                         STATUS   ROLES                  AGE   VERSION
    ip-127-0-151-14.cloud.compute.intra          Ready    worker                 23h   v1.26.6+73ac561
    ip-127-0-159-95.cloud.compute.intra          Ready    control-plane,master   23h   v1.26.6+73ac561
    ip-127-0-176-215.cloud.compute.intra         Ready    control-plane,master   23h   v1.26.6+73ac561
    ip-127-0-190-238.cloud.compute.intra         Ready    worker                 23h   v1.26.6+73ac561
    ip-127-0-197-202.cloud.compute.intra         Ready    worker                 23h   v1.26.6+73ac561
    ip-127-0-207-151.cloud.compute.intra         Ready    control-plane,master   23h   v1.26.6+73ac561
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.