How to ensure pods are evenly scheduled across nodes in OpenShift 4?

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Scheduler

Issue

  • Pods are not evenly distributed across all the nodes.
  • How can pods be scheduled evenly on all the nodes in the cluster ?

Resolution

It is possible to evenly distribute pods from "pod group" across nodes via pod topology spread constraints (with topologySpreadConstraints field from pods, deployments and other CRs that defines pods). For additional information refer to controlling pod placement by using pod topology spread constraints.

This allow users to control how pods are distributed across different topologies (e.g., nodes, zones, etc.).

Structure of the topologySpreadConstraints field

The structure and fields of the topologySpreadConstraints is as follows:

spec:
  topologySpreadConstraints:
  - maxSkew: 1 # [1]
    topologyKey: node # [2]
    whenUnsatisfiable: DoNotSchedule # [3]
    labelSelector: # [4]
      matchLabels:
        foo: bar # [5]

[1] The maximum difference in the number of pods between any two topology domains. The default is 1, and you cannot specify a value of 0.
[2] The key of a node label. Nodes with this key and identical value are considered to be in the same topology.
[3] How to handle a pod if it does not satisfy the spread constraint. The default is DoNotSchedule, which tells the scheduler not to schedule the pod. Set to ScheduleAnyway to still schedule the pod, but the scheduler prioritizes honoring the skew to not make the cluster more imbalanced.
[4] Pods that match this label selector are counted and recognized as a group when spreading to satisfy the constraint. Be sure to specify a label selector, otherwise, no pods can be matched.
[5] Be sure that this Pod spec also sets its labels to match this label selector if you want it to be counted properly in the future.

Refer to example configurations for pod topology spread constraints for additional information.

Root Cause

Pods get scheduled on the node with the highest score, and this can lead to an uneven distribution of pods across nodes in the RHOCP cluster. For additional information about pod scheduling, refer to pods are not getting allocated evenly on worker nodes in OpenShift.

Using the pod topology spread constraints feature (via the topologySpreadConstraints field from pods, deployments and other CRs that defines pods), it is possible to distribute new scaled pods from a pod or deployment CR across nodes.

  • Below are the 13 predicates of filtering and scoring scheduler refers:

    FILTER |== (filter valid nodes)
  PodFitsHostPorts
  PodFitsHost
  PodFitsResources
  PodMatchNodeSelector
  NoVolumeZoneConflict
  NoDiskConflict
  MaxCSIVolumeCount
  CheckNodeMemPressure
  CheckNodePIDPressure
  CheckNodeDiskPressure
  CheckNodeCondition
  PodToloratesNodeTaints
  CheckVolumeBinding
SCORE |== (Score nodes from best to worse)
  SelectorSpreadPriority
  InterPodAffinityPriority
  LeastRequestPriority
  MostRequestPriority
  RequestedToCapacityRatioPriority
  BalancedResourceAllocation
  NodePreferAvoidPodsPriority
  NodeAffinityPriority
  TaintTolerationPriority
  ImageLocalityPriority
  ServiceSpreadingPriority
  EqualPriority
  EvenPodsSpreadPriority

Diagnostic Steps

  • Check the scheduled pod logs:

    2025-02-12T19:45.77179304 I0212 19:45:05.771783 1 schedule_one.go:763] "Calculated node's final score for pod" pod="project-name/pod-name" node="node-name" score=619
2025-02-12T19:45.77179304 I0212 19:45:05.771790 1 schedule_one.go:763] "Calculated node's final score for pod" pod="project-name/pod-name" node="node-name" score=616
2025-02-12T19:45.77181854 I0212 19:45:05.771809 1 schedule_one.go:763] "Calculated node's final score for pod" pod="project-name/pod-name" node="node-name" score=623
2025-02-12T19:45.77205008 I0212 19:45:05.772029 1 default_binder.go:53] "Attempting to bind pod to node" pod="project-name/pod-name" node="node-name"

    Refer to how to find the scheduler decisions in OpenShift 4? for additional information

  • The Scheduler scheduled this pod on the node having the highest score, i.e. 624.

  • Check number of nodes in the cluster :

        $ oc get nodes -l node-role.kubernetes.io/worker=,topology.kubernetes.io/zone=ap-south-1a
    NAME                                         STATUS   ROLES    AGE    VERSION
    node1.ap-south-1.compute.internal   Ready    worker   6h3m   v1.27.8+4fab27b

    $ oc get nodes -l node-role.kubernetes.io/worker=,topology.kubernetes.io/zone=ap-south-1b
    NAME                                         STATUS   ROLES    AGE    VERSION
    node2.ap-south-1.compute.internal     Ready    worker   6h2m   v1.27.8+4fab27b

    $ oc get nodes -l node-role.kubernetes.io/worker=,topology.kubernetes.io/zone=ap-south-1c
    NAME                                         STATUS   ROLES    AGE    VERSION
    node3.ap-south-1.compute.internal   Ready    worker   2d     v1.27.8+4fab27b

  • Create a project and deployment for testing:

        $ oc new-project topsep

    $ cat dep.yaml 
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: test
      namespace: topsep
    spec:
      replicas: 9
      selector:
        matchLabels:
          app: test
      template:
        metadata:
          labels:
            app: test
        spec:
          containers:
          - image: image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest
            imagePullPolicy: Always
            name: container
            ports:
            - containerPort: 8080
              protocol: TCP
            resources: {}
          topologySpreadConstraints:
          - maxSkew: 3
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                app: test
          - maxSkew: 3
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                app: test

    $ oc apply -f dep.yaml

  • Verify if the pods are evenly distributed across the nodes:

    $ for i in $(oc get nodes -oname -l node-role.kubernetes.io/worker= | awk -F/ '{ print $2 }'); do oc get pods -owide | grep $i  ; done  --> to list the pods
$ for i in $(oc get nodes -oname -l node-role.kubernetes.io/worker= | awk -F/ '{ print $2 }'); do oc get pods -owide | grep $i | wc -l; done --> to check the number of pods
3
3
3
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.