How to renew etcd certificates in OpenShift 4 when not yet expired?

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4

Issue

  • How to renew the etcd certificates in OpenShift 4 when the certificates are not yet expired?
  • How to rotate the etcd certificates?
  • The etcd certificates in OpenShift 4 are not automatically rotated.

Resolution

This procedure is usually not needed in OpenShift Container Platform 4.9 and newer releases as the etcd certificates are rotated automatically starting with OpenShift 4.9. Check the Root Cause section for additional information.

Note: This guide only works when the certificates are not yet expired. If etcd certificates have already expired, refer to How to renew etcd certificates in OpenShift 4 when are already expired.

Prerequisites

Refresh Steps

1. Check if the operator is running and the status of the operator:

$ oc get co etcd

NAME   VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
etcd   4.8.57    True        False         False      20d

Make sure that the operator is not Degraded or Progressing. If the operator is Degraded, do not proceed.

$ oc get co etcd -o json | jq -r '.status.conditions[] | select(.type=="Available")'

{
  "lastTransitionTime": "2023-07-11T23:34:57Z",
  "message": "StaticPodsAvailable: 3 nodes are active; 3 nodes are at revision 7\nEtcdMembersAvailable: 3 members are available",
  "reason": "AsExpected",
  "status": "True",
  "type": "Available"
}

Make sure that the condition Available is True and AsExpected.

2. Verify if the certificates will expire soon:

$ (echo -e "SECRET_NAME\tEXPIRATION_DATE" && oc get secret -n openshift-etcd -o json | jq -r '.items[] | select(( (.metadata.name|startswith("etcd-peer")) or (.metadata.name|startswith("etcd-serving")) ) and .type=="kubernetes.io/tls") | [.metadata.name,.data."tls.crt"] | @tsv' | while read name cert; do echo -en "$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate | sed 's/notAfter=//g' ; done ) | column -t -s $'\t'

3. Make a backup of the secrets:

$ oc get secret -o yaml -n openshift-etcd $(oc get -n openshift-etcd secret -o jsonpath='{range .items[?(.type=="kubernetes.io/tls")]}{.metadata.name}{"\n"}{end}' | grep -e "etcd-peer\|etcd-serving") > etcd-certs-backup.yaml

4. Delete the secrets:
Make sure to delete all secrets in one command. Do not delete the secrets one by one as it could cause issue with the ETCD cluster operator.

$ oc delete secret -n openshift-etcd $(oc get -n openshift-etcd secret -o jsonpath='{range .items[?(.type=="kubernetes.io/tls")]}{.metadata.name}{"\n"}{end}' | grep -e "etcd-peer\|etcd-serving")

5. Check if the new secrets are created:

$ oc get secret -n openshift-etcd $(oc get -n openshift-etcd secret -o jsonpath='{range .items[?(.type=="kubernetes.io/tls")]}{.metadata.name}{"\n"}{end}' | grep -e "etcd-peer\|etcd-serving")

Attention: Do not proceed if the secrets are not recreated! If not, please move to section "Recovery".

6. Verify that new certificates are refreshed:

(echo -e "SECRET_NAME\tEXPIRATION_DATE" && oc get secret -n openshift-etcd -o json | jq -r '.items[] | select(( (.metadata.name|startswith("etcd-peer")) or (.metadata.name|startswith("etcd-serving")) ) and .type=="kubernetes.io/tls") | [.metadata.name,.data."tls.crt"] | @tsv' | while read name cert; do echo -en "$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate | sed 's/notAfter=//g' ; done ) | column -t -s $'\t'

7. Watch the operator and the pods while applying the changes automatically:

$ watch -n 5 'oc get co etcd; echo; oc get pods -n openshift-etcd'

Recovery Steps

If the new secrets are not being generated, create the secrets back from the backup file:

$ oc create -f etcd-certs-backup.yaml

In case of any issue, please This content is not included.open a Support Case with Red Hat Support Team.

Root Cause

The etcd certificates are rotated automatically starting with OpenShift 4.9 as shown in the OpenShift Container Platform 4.9 release notes - Automatic rotation of etcd certificates, but not in OpenShift 4.8 and older releases.

Diagnostic Steps

Verify the expiration dates of the etcd certificates:

$ (echo -e "SECRET_NAME\tEXPIRATION_DATE" && oc get secret -n openshift-etcd -o json | jq -r '.items[] | select(( (.metadata.name|startswith("etcd-peer")) or (.metadata.name|startswith("etcd-serving")) ) and .type=="kubernetes.io/tls") | [.metadata.name,.data."tls.crt"] | @tsv' | while read name cert; do echo -en "$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate | sed 's/notAfter=//g' ; done ) | column -t -s $'\t'

### expected output similar to:
SECRET_NAME                                EXPIRATION_DATE
etcd-peer-master-0.example.com             Feb 16 19:38:15 2027 GMT
etcd-peer-master-1.example.com             Feb 16 19:38:14 2027 GMT
etcd-peer-master-2.example.com             Feb 16 19:38:14 2027 GMT
etcd-serving-master-0.example.com          Feb 16 19:38:15 2027 GMT
etcd-serving-master-1.example.com          Feb 16 19:38:14 2027 GMT
etcd-serving-master-2.example.com          Feb 16 19:38:13 2027 GMT
etcd-serving-metrics-master-0.example.com  Feb 16 19:38:15 2027 GMT
etcd-serving-metrics-master-1.example.com  Feb 16 19:38:15 2027 GMT
etcd-serving-metrics-master-2.example.com  Feb 16 19:38:14 2027 GMT

Verify the certificates served by the etcd members:

$ for each in $(oc get pods -n openshift-etcd -l app=etcd -o jsonpath='{.items[*].status.podIP}{"\n"}'); do echo "ETCD member https://$each:2379"; echo -n |openssl s_client -connect $each:2379 -showcerts 2>/dev/null|openssl x509 -noout -dates; echo "----"; done

### expected output similar to:
ETCD member https://10.0.1.1:2379
notBefore=Jul 22 19:46:37 2023 GMT
notAfter=Jul 21 19:46:38 2026 GMT
----
ETCD member https://10.0.1.2:2379
notBefore=Jul 22 19:46:37 2023 GMT
notAfter=Jul 21 19:46:38 2026 GMT
----
ETCD member https://10.0.1.3:2379
notBefore=Jul 22 19:46:38 2023 GMT
notAfter=Jul 21 19:46:39 2026 GMT
----
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.