Pods are not re-distributed evenly across all applicable nodes after adding or starting worker/compute nodes in OpenShift

Environment

• Red Hat OpenShift Container Platform (OCP)
  • 3
  • 4
• Existing workloads
• After adding a new worker or compute node

Issue

• Only newly deployed pods are being scheduled to the new worker.
• Pods are not evenly scheduled across nodes.

Resolution

Pods that are already running are not automatically rescheduled to other nodes.

It is possible to use the Descheduler to reallocate pods in nodes with free resources. Configuration of this feature may meet some or all of one's pod re-distribution needs. Refer to deploying and configuring the Kube Descheduler Operator on OpenShift for installation and configuration example.

Refer to Pods are not getting allocated evenly on worker nodes in OpenShift for additional information about how pods are scheduled in nodes.

Root Cause

Scheduler logic is only applied when a pod is newly deployed. So for example if adding a new compute node (or a node is being rebooted) there is no logic that will "re-schedule" an existing pod running on a given compute node to now run on the newly added compute node in order to evenly distribute the pods. The new node will be considered when new pods are deployed however.

The Descheduler will scale down running applications on loaded nodes so they can be re-scheduled on other nodes.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.