Labeling OpenShift Data Foundation (ODF) nodes as infra nodes

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4
  • Red Hat OpenShift Data Foundation (RHODF) 4

Issue

  • How to separate application and storage workloads and avoid double use of OCP and ODF subscriptions for nodes that are used exclusively for storage services.

Resolution

  • Label the ODF nodes to add the infra label and annotate the odf namespace with a default node selector:
oc label nodes <NodeName> node-role.kubernetes.io/infra=''
oc label nodes <NodeName> node-role.kubernetes.io=infra
oc annotate namespace openshift-storage openshift.io/node-selector=

Delete the CSI pods in Pending state so new pods will get scheduled:

oc delete pod -l app=csi-cephfsplugin -n openshift-storage
oc delete pod -l app=csi-rbdplugin -n openshift-storage

Please note that some of the CSI plugin pods will continue running on worker nodes. The CSI plugin pods need to run on every node where storage needs to be provisioned. Other pods like rook-ceph-tools may also run on worker nodes.

  • To have dedicated storage nodes, where other pods are prevented from being scheduled on the same infra nodes, add a taint:
oc adm taint nodes -l node-role.kubernetes.io/infra node.ocs.openshift.io/storage=true:NoSchedule
  • If multiple unique nodes are labeled as part of infra (storage, worker, router, etc.), all these nodes should belong to a single Machine Config Pool (MCP) named infra, else this error may appear during updates:
can't get pool for node "infra-node-1": node infra-node-1 belongs to 2 custom roles, cannot proceed with this Node

  • For more information on configuring infrastructure nodes, refer to the solutions article Infrastructure Nodes in OpenShift 4.

  • Note that, infrastructure nodes for ODF and OCP can be shared if the nodes have enough compute resources. However, only infrastructure related pods are allowed to be run on infrastructure nodes.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.