Distributing pod replicas evenly on available nodes using "PodAntiAffinity"

Environment

• Red Hat OpenShift Container Platform (RHOCP) - 4.x
• Red Hat OpenShift Service on AWS (ROSA)
• Azure Red Hat OpenShift (ARO)

Issue

• How to configure deployment to place pod replicas on different nodes?
• How to ensure the scheduler does not co-locate replicas on a single node?
• How to distribute pod replicas evenly on available nodes?
• How to balance pods across available nodes to limit impact if one node goes down?

Resolution

This can be achieved by using "PodAntiAffinity" in the deployment itself:

Example:

The below YAML snippet represents a simple redis deployment with three replicas and selector label app=store. The deployment has PodAntiAffinity configured to ensure the scheduler does not co-locate replicas on a single node.

oc get deployment redis-cache -n <namespace> -o yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-cache
spec:
  selector:
    matchLabels:
      app: store
  replicas: 3
  template:
    metadata:
      labels:
        app: store
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - store
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: redis-server
        image: redis:3.2-alpine

Root Cause

• The default OpenShift Container Platform pod scheduler is responsible for determining placement of new pods onto nodes within the cluster. It reads data from the pod and tries to find a node that is a good fit based on configured policies. It is completely independent and exists as a standalone/pluggable solution. It does not modify the pod and just creates a binding for the pod that ties the pod to the particular node.

• The anti-affinity rule was configured to preferredDuringSchedulingIgnoredDuringExecution, so it was possible to have several pods on the same node.

Diagnostic Steps

Check anti-affinity in the deployment file:

oc get deployment <deployment-name> -n <namespace> -o yaml

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.