Understand Kubelet CA cert auto renewal in Red Hat OpenShift 4

Background

• Openshift v4 automatically generates a new kube-apiserver-to-kubelet-signer CA certificates at 292 days and removes old CA certificate after 365 days.
• The renewal of cert as well as removal of certs causes rolling reboot of nodes by Content from github.com is not included.machine-config-operator.
• The reboot is performed as a rolling reboot and is designed to be low impact and should not impact the cluster workloads. Assuming the workloads have been designed to accommodate kubernetes principals.
• Recommendation - Upgrade cluster to version This content is not included.4.7.4 to avail the fix to avoid the node reboots during automatic CA certificate renewal.

Checking the validity of the certs in CA bundle

• The CA bundle will generally have a single cert in it which is active currently.

• After a CA cert renewal takes place, the CA bundle will also have an old cert in it which is still present because it has not expired and been removed yet. Once it expires, it will be removed automatically causing a separate rolling reboot of nodes as the rendered-* machineconfigs will be created again to reflect on all nodes in the respective master and worker pools.

• To fetch the CA bundle the following command can be used:

$ oc get cm -n openshift-kube-apiserver-operator kube-apiserver-to-kubelet-client-ca -ojsonpath='{.data.ca-bundle\.crt}'

• To check the issuer and validity of the certs inside the CA bundle, the given command can be run:

$ oc get cm -n openshift-kube-apiserver-operator kube-apiserver-to-kubelet-client-ca -ojsonpath='{.data.ca-bundle\.crt}'| openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -text | egrep -A4 Issuer
        Issuer: CN=openshift-kube-apiserver-operator_kube-apiserver-to-kubelet-signer@1605510924
        Validity
            Not Before: Nov 16 07:15:24 2020 GMT
            Not After : Nov 16 07:15:25 2021 GMT
        Subject: CN=openshift-kube-apiserver-operator_kube-apiserver-to-kubelet-signer@1605510924
--
        Issuer: OU=openshift, CN=kube-apiserver-to-kubelet-signer
        Validity
            Not Before: Jan 29 07:15:17 2020 GMT
            Not After : Jan 28 07:15:17 2021 GMT
        Subject: OU=openshift, CN=kube-apiserver-to-kubelet-signer

Checking the node reboot time

• This check is simply meant to know the uptime of all the nodes to confirm if the cluster wide CA cert renewal change has happened or not.
• The expected output here is to see that every node has rebooted at around the same time with few minutes gap due to the rolling reboot.

$ for n in $(oc get node -oname); do echo "------ $n -------"; oc debug --quiet $n -- uptime -s; done

What is next?

The Openshift product engineering team has acknowledged the concern of node reboots during automatic CA certificate renewal and is working towards improving and optimizing it as a future Openshift feature (subject to change).

How many times will a cluster perform node reboots with regards to automatic CA certificate renewal?

• The kube-apiserver-to-kubelet-signer CA certificate has a validity of 1 year(365 days) and is renewed at 80% of it's expiry at 292 days. This is the first set of automatic rolling reboots that will happen.
• After the CA cert renewal is done, the original/previous CA certificate is not deleted and still has validity for another 73 days till it finishes 365 days. Once it is expired, it will be removed automatically which will cause the rolling reboot of nodes again.
• The above two sets of automatic node reboots will happen for each individual CA certificate so need to calculate the next date after every renewal.
• As an example, considering the below command output which shows the issuer and the validity dates of the new CA cert and the old CA cert:

$ oc get cm -n openshift-kube-apiserver-operator kube-apiserver-to-kubelet-client-ca -ojsonpath='{.data.ca-bundle\.crt}'| openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -text | egrep -A4 Issuer
        Issuer: CN=openshift-kube-apiserver-operator_kube-apiserver-to-kubelet-signer@1605510924
        Validity
            Not Before: Nov 16 07:15:24 2020 GMT
            Not After : Nov 16 07:15:25 2021 GMT
        Subject: CN=openshift-kube-apiserver-operator_kube-apiserver-to-kubelet-signer@1605510924
--
        Issuer: OU=openshift, CN=kube-apiserver-to-kubelet-signer
        Validity
            Not Before: Jan 29 07:15:17 2020 GMT
            Not After : Jan 28 07:15:17 2021 GMT
        Subject: OU=openshift, CN=kube-apiserver-to-kubelet-signer

• The renewal time of new certificate after 292 days considering 'GMT' timezone can be calculated as follows after picking the "Valid from" date of new cert:

$ TZ=GMT date -d "Nov 16 07:15:24 2020 GMT+292 days"
Sat Sep  4 07:15:24 GMT 2021
$

• The removal time of old certificate can be known from the "Not After :" date field which is "Jan 28 07:15:17 2021 GMT".

What can customers do meanwhile?

• The customers are advised to upgrade their clusters to atleast This content is not included.4.7.4 in order to avail the fix of having no disruption via reboots during automatic CA certificate renewal.
• The above troubleshooting commands can be referred to know when the cluster's CA certs would be renewed and removed again which would cause automatic rolling reboot of nodes.
• Additionally, the customers are also advised to increase the availability of their applications by increasing the number of replicas in the deployment or can fine tune an application's disruption using Content from kubernetes.io is not included.PodDisruptionBudget(PDB) as per best practices.

How can CA certificate renewal be initiated ahead of schedule?

• Customers are advised to upgrade their cluster to at least This content is not included.4.7.4 as the preferred and least disruptive solution.
• However, for customers that cannot upgrade their cluster before the cluster's CA renewal date, and who wish to have some control over the time when the renewal takes place, it is possible to initiate the renewal ahead of time.
• Initiating the renewal ahead of time can only be performed by users that are members of the cluster-admin role.
  • For an OpenShift Dedicated clusters (OSD), users with the This page is not included, but the link has been rewritten to point to the nearest parent document.dedicated-admin role can also initiate the renewal, but only for OSD clusters that are on versions 4.5.x or 4.6.x.
• Renewal can be initiated via the following command:

$ oc annotate -n openshift-kube-apiserver-operator secret kube-apiserver-to-kubelet-signer auth.openshift.io/certificate-not-after-

• Please note that the cluster will commence an immediate rolling reboot of nodes to apply the renewed CA certificate after the issuing of this command.

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.