Checking etcd certificate expiry in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • etcd certificates

Issue

  • The etcd certificates are not automatically rotated/refreshed in OpenShift versions before 4.9, causing cluster-wide outage when expired.
  • How to check etcd certificate expiry and renew them before expiry?
  • How to check etcd certificate expiry before shutting down an OpenShift cluster.

Resolution

The certificates are managed by the operator, they can be automatically recreated, but they are not automatically rotated before expiration in versions before OpenShift 4.9. The etcd certificates are stored as secrets in the openshift-etcd project.

The following command can be used to extract the certificates and examine their end date:

$ (echo -e "SECRET_NAME\tEXPIRATION_DATE" && oc get secret -n openshift-etcd -o json | jq -r '.items[] | select(( (.metadata.name|startswith("etcd-peer")) or (.metadata.name|startswith("etcd-serving")) ) and .type=="kubernetes.io/tls") | [.metadata.name,.data."tls.crt"] | @tsv' | while read name cert; do echo -en "$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate | sed 's/notAfter=//g' ; done ) | column -t -s $'\t'

### expected output similar to:
SECRET_NAME                                EXPIRATION_DATE
etcd-peer-master-0.example.com             Feb 16 19:38:15 2027 GMT
etcd-peer-master-1.example.com             Feb 16 19:38:14 2027 GMT
etcd-peer-master-2.example.com             Feb 16 19:38:14 2027 GMT
etcd-serving-master-0.example.com          Feb 16 19:38:15 2027 GMT
etcd-serving-master-1.example.com          Feb 16 19:38:14 2027 GMT
etcd-serving-master-2.example.com          Feb 16 19:38:13 2027 GMT
etcd-serving-metrics-master-0.example.com  Feb 16 19:38:15 2027 GMT
etcd-serving-metrics-master-1.example.com  Feb 16 19:38:15 2027 GMT
etcd-serving-metrics-master-2.example.com  Feb 16 19:38:14 2027 GMT

If the OpenShift cluster is going to be shut down (or a version before OpenShift 4.9 is still in use) renew the certificates as explained in How to renew etcd certificates in OpenShift 4 when not yet expired.

If the certificates are already expired, please check How to renew etcd certificates in OpenShift 4.8 and lower when certificates are already expired.

Root Cause

When a cluster is first installed, the TLS certs used for cluster communication are generated and stored by the etcd service with a fixed expiration date. Prior to OpenShift 4.9, certificates are not auto rotated. This means that once a certificate expires, communication within the cluster is no longer possible.

etcd certificate auto rotation was introduced in OCP version 4.9. For more details on this new feature see the OpenShift Container Platform 4.9 release notes - Automatic rotation of etcd certificates.

Diagnostic Steps

The following command can be used to extract the certificates and examine their end date:

$ (echo -e "SECRET_NAME\tEXPIRATION_DATE" && oc get secret -n openshift-etcd -o json | jq -r '.items[] | select(( (.metadata.name|startswith("etcd-peer")) or (.metadata.name|startswith("etcd-serving")) ) and .type=="kubernetes.io/tls") | [.metadata.name,.data."tls.crt"] | @tsv' | while read name cert; do echo -en "$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate | sed 's/notAfter=//g' ; done ) | column -t -s $'\t'
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.