Node went to NotReady due to the network issue in RHOCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4

Issue

• Newly added node went to NotReady state and showing the below error message:

  container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: 
  No CNI configuration file in /etc/kubernetes/cni/net.d/
  

• Pods of the openshift-multus and openshift-network-diagnostics namespace are not able to schedule on the NotReady node.

Resolution

For OCP 4.13 and below:

• To resolve this issue restart the ovnkube-master pods:

  $ for pod in $(oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-master -o name)
  do
      oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-master
      oc delete ${pod} -n openshift-ovn-kubernetes
      sleep 100
  done
  $ oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-master
  NAME                   READY   STATUS    RESTARTS   AGE
  ovnkube-master-xxxxx   6/6     Running   0          5m17s
  ovnkube-master-xxxxx   6/6     Running   0          3m33s
  ovnkube-master-xxxxx   6/6     Running   0          108s
  

For OCP 4.14 and above:

• To resolve this issue restart the ovnkube-control-plane pods:

  $ for pod in $(oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-control-plane -o name)
  do
      oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-control-plane
      oc delete ${pod} -n openshift-ovn-kubernetes
      sleep 5
  done
  $ oc get pod -n openshift-ovn-kubernetes -l app=ovnkube-control-plane
    ovnkube-control-plane-xxxxxxxxxx-xxxxx                        2/2     Running     0          19s
    ovnkube-control-plane-xxxxxxxxxx-xxxxx                        2/2     Running     0          13s
  

Diagnostic Steps

• Verify the status of Nodes and network co:

  $ oc  get  nodes | grep -i notready
  workernode01   NotReady   worker    25m    v1.24.11+af0420d
  

  $ oc get co network
  network       4.11.33   True        True          True       19m
  

• Check the description of NotReady node:

  $ oc  describe node/<node-name>
  lastTransitionTime: "2023-04-17T13:15:13Z"
  message: DaemonSet "/openshift-ovn-kubernetes/ovnkube-node" rollout is not making
     progress - last change 2023-04-17T13:04:53Z
  reason: RolloutHung
  container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI 
  configuration file in /etc/kubernetes/cni/net.d/
  

• Verify the pods on the NotReady node:

  $ oc get pod -A | grep -e 'Pending'
  openshift-multus                                   network-metrics-daemon-xxx                                                        0/2     Pending             0          29m                     
  workernode01
  openshift-network-diagnostics                      network-check-target-yyy                                                           0/1     ContainerCreating   0          31m                     
  workernode01
  

• Verify the journalctl logs of node and look for below error:

  worker not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xjch7" podUID=db0b0c51-f412-4bd3-ad99-dc5d0cdd4b8e
  Apr 17 13:53:52 Workernode01.com hyperkube[5347]: E0417 13:53:52.157536    5347 pod_workers.go:965] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xxx"podUID=ae8e5c90-a4af-485d-b11a-634e4b51b1f4
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.