Unable to connect to the server: EOF - never ending certification approvals

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4

Issue

• Users get error Unable to connect to the server: EOF while oc login.
• CSR getting generated every 10 minutes
• As soon as certificates are approved, new ones are issued within a couple of minutes.

Resolution

• While oc login does not work, the first step here would be to connect to master nodes using ssh and getting access to oc command as per the steps mentioned in the article.

• Once the access is available to oc command, clean up all CSRs.

  $ oc delete csr --all
  

• Delete the bad certificate

  $ oc delete secret -n openshift-config-managed kube-controller-manager-client-cert-key
  

• Then approve the CSRs as they come in and verify that they are now "Approved,Issued"

      $ oc get csr -o name | xargs oc adm certificate approve 
  
      $ oc get csr -A
  

• If etcd and kube-scheduler is broken patch them.

  $ oc patch etcd cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
  

  $ oc patch kubescheduler cluster -p='{"spec": {"forceRedeploymentReason": "recovery-'"$( date --rfc-3339=ns )"'"}}' --type=merge
  

Root Cause

• Certificate is either expired or corrupt.
• kube-controller-manager might be broken due to which csr was not getting "Approved, Issued".
• etcd and kube-scheduler pod having some issue.

Diagnostic Steps

• Check the pending CSR.

  $ oc get CSR -A 
  

• Check the kube-controller-manager logs.

  2022-06-27T06:33:31.318347695Z E0627 06:33:31.318207       1 leaderelection.go:330] error retrieving resource 
  lock kube-system/kube-controller-manager: Unauthorized
  2022-06-27T06:33:36.359141076Z E0627 06:33:36.359093       1 leaderelection.go:330] error retrieving resource 
  lock kube-system/kube-controller-manager: Unauthorized
  2022-06-27T06:33:41.798588413Z E0627 06:33:41.798549       1 leaderelection.go:330] error retrieving resource 
  lock kube-system/kube-controller-manager: Unauthorized
  

• Check the ETCD pods health

  $ oc get pods -n openshift-etcd
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.