Openshift-dns daemonsets doesn't include toleration to run on nodes with taints

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4

Issue

• Openshift-dns daemonset doesn't include toleration to run on nodes with taints. After a NoSchedule taint is configured for a node, the daemon set stops managing the pods on that node and 2 things happen:

  • If the pods are deleted on nodes with taint, they won't be recovered.
  • no dns-default pod found for node is reported.
  • The following alerts are triggering:

  Pods of `DaemonSet openshift-dns/dns-default` are running where they are not supposed to run.
  

Resolution

• Add the below toleration by editing the dns operator as below - NOTE: By adding such toleration all Pods from all nodes will get a restart. This is a harmless operation. If unsure, please raise a case by following the How do I open and manage a support case on the Customer Portal? article.

$ oc edit dns.operator/default

apiVersion: operator.openshift.io/v1
kind: DNS
metadata:
  name: default
spec:
  nodePlacement:
    tolerations:
    - operator: Exists

• Another way is to delete the dns-default Pod on the Node when configured not to run the DNS Pod on the node where the taint is configured.

Diagnostic Steps

• Check desired nodes for the dns deamonsets:

  $ oc -n openshift-dns get ds 
  NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
  dns-default     5         5         5       5            5           kubernetes.io/os=linux   5d3h
  

• Apply NoSchedule taint to node:

  $ oc adm taint nodes worker-1 node-role.kubernetes.io/infra:NoSchedule
  node/worker-1 tainted
  $ oc adm taint nodes worker-2 node-role.kubernetes.io/infra:NoSchedule
  node/worker-2 tainted
  

• Check that the desired count has less than the number of nodes:

  $ oc -n openshift-dns get ds 
  NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
  dns-default     3         3         3       3            3           kubernetes.io/os=linux   5d3h
  

• This error is displayed in the OpenShift console:

  Pods of DaemonSet openshift-dns/dns-default are running where they are not supposed to run
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.