How to check the internal certificates expiry in OpenShift Container Platform 4?

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4.x

Issue

  • How to check the internal certificates validity information in RHOCP 4.x ?
  • What is the location of control plane and node certificates in RHOCP 4.x ?

Resolution

  • Unlike RHOCP 3.x, all the internal certificates are managed by the cluster itself and through the respective operators. The various certificate-key pairs are distributed through secrets and config maps in RHOCP 4.x

  • The server-client certificates used by the control plane components can be checked as follows:

1 . API:

  • The serving cert and key pair used by both internal and external API are stored inside the secrets in the namespace openshift-kube-apiserver. So the validity of those can be retrieved as:

  • External API:

$ oc get secret -n openshift-kube-apiserver external-loadbalancer-serving-certkey -o yaml -o=custom-columns=":.data.tls\.crt" | tail -1 | base64 -d | openssl x509 -noout -dates
  • Internal API:
$ oc get secret -n openshift-kube-apiserver internal-loadbalancer-serving-certkey -o yaml -o=custom-columns=":.data.tls\.crt" | tail -1 | base64 -d | openssl x509 -noout -dates
  • The same cert-key pair are stored in each master node, in the location /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets and the validity can be checked as:
$ ssh core@master_hostname
$ sudo -i
$ openssl x509 -in  /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/external-loadbalancer-serving-certkey/tls.crt -noout -dates
$ openssl x509 -in  /etc/kubernetes/static-pod-resources/kube-apiserver-certs/secrets/internal-loadbalancer-serving-certkey/tls.crt -noout -dates
  • So either from secret or the cert-key pair file locations inside the master nodes, we can get the certificate information.

2 . Kube Controller Manager:

  • Client certificate:
$ oc get secret kube-controller-manager-client-cert-key -n openshift-kube-controller-manager -o=custom-columns=":.data.tls\.crt" | tail -1 | base64 -d | openssl x509 -noout -dates

OR

$ openssl x509 -in /etc/kubernetes/static-pod-resources/kube-controller-manager-certs/secrets/kube-controller-manager-client-cert-key/tls.crt -noout -dates
  • Server certificate:
$ oc get secret serving-cert -n openshift-kube-controller-manager -o=custom-columns=":.data.tls\.crt" | tail -1 | base64 -d | openssl x509 -noout -dates

OR

$ openssl x509 -in /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-xy/secrets/serving-cert/tls.crt -noout -dates

3 . Kube Scheduler:

  • Client certificate:
$ oc get secret kube-scheduler-client-cert-key -n openshift-kube-scheduler -o=custom-columns=":.data.tls\.crt" | tail -1 | base64 -d | openssl x509 -noout -dates

OR

$ openssl x509 -in /etc/kubernetes/static-pod-resources/kube-scheduler-certs/secrets/kube-scheduler-client-cert-key/tls.crt -noout -dates
  • Server certificate:
$ oc get secret serving-cert -n openshift-kube-scheduler -o=custom-columns=":.data.tls\.crt" | tail -1 | base64 -d | openssl x509 -noout -dates

OR

$ openssl x509 -in /etc/kubernetes/static-pod-resources/kube-scheduler-pod-xy/secrets/serving-cert/tls.crt -noout -dates

4 . ETCD Certificates:

  • Go to the openshift-etcd project and get the tls secrets:
$ oc project openshift-etcd
$ oc get secrets | grep tls
  • There are following 3 sets of secrets for each master node/host,which holds the server,peer and metric certificate and key pairs:
etcd-peer-<master_hostname> 
etcd-serving-<master_hostname> 
etcd-serving-metrics-<master_hostname>

a . The etcd-peer certificate is used for the etcd peer-to-peer communication.
b . The etcd-serving certificate is used as the serving certificate by each etcd host.
c . The etcd-serving-metrics certificate is used for getting the etcd metrics.

  • ETCD peer cert:
$ oc get secret etcd-peer-<master_hostname>  -o=custom-columns=":.data.tls\.crt" | tail -1 | base64 -d | openssl x509 -noout -dates
  • ETCD Serving cert:
$ oc get secret etcd-serving-<master_hostname>  -o=custom-columns=":.data.tls\.crt" | tail -1 | base64 -d | openssl x509 -noout -dates
  • ETCD Serving Metrics Cert: 
$ oc get secret etcd-serving-metrics-<master_hostname> -o=custom-columns=":.data.tls\.crt" | tail -1 | base64 -d | openssl x509 -noout -dates
  • The above 3 certs can also be checked directly from master nodes as:
$ ssh core@master_hostname
$ sudo -i
RHOCP versions prior to 4.11
  • ETCD Serving cert:
$ for i in /etc/kubernetes/static-pod-resources/etcd-certs/secrets/etcd-all-serving/*.crt; do echo $i; openssl x509 -in $i -noout -dates; done
  • ETCD peer cert:
$ for i in /etc/kubernetes/static-pod-resources/etcd-certs/secrets/etcd-all-peer/*.crt; do echo $i; openssl x509 -in $i -noout -dates; done
  • ETCD Serving Metrics Cert:
$ for i in /etc/kubernetes/static-pod-resources/etcd-certs/secrets/etcd-all-serving-metrics/*.crt; do echo $i; openssl x509 -in $i -noout -dates; done
RHOCP versions 4.11 and above
  • ETCD Serving cert:
$ for i in /etc/kubernetes/static-pod-resources/etcd-certs/secrets/etcd-all-certs/etcd-serving-*.crt; do echo $i; openssl x509 -in $i -noout -dates; done
  • ETCD peer cert:
$ for i in /etc/kubernetes/static-pod-resources/etcd-certs/secrets/etcd-all-certs/etcd-peer*.crt; do echo $i; openssl x509 -in $i -noout -dates; done
  • ETCD Serving Metrics Cert:
$ for i in /etc/kubernetes/static-pod-resources/etcd-certs/secrets/etcd-all-certs/etcd-serving-metrics*.crt; do echo $i; openssl x509 -in $i -noout -dates; done

5 . Node Certificates:

  • Each node has a /var/lib/kubelet/pki directory, which holds the server and client certs for the particular node(kubelet).

a . kubelet-client-current.pem = Which is used as the kubelet client cert.
b . kubelet-server-current.pem = Which is used as the kubelet server cert.

There are other PEM files that are already rotated certs and also the symlinks of the above 2 certificates

  • To check the server and client certs used by the kubelet:
$ for cert in /var/lib/kubelet/pki/kubelet-{client,server}-current.pem; do echo $cert; openssl x509 -in $cert -noout -dates; done

Can also be checked individually as:

$ openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -dates

$ openssl x509 -in /var/lib/kubelet/pki/kubelet-server-current.pem -noout -dates

6 . Ingress certificate:

  • Used by the ingress router and all the secured routes are using this certificate, unless a cert-key pair is explicitly provided through the route YAML.

  • To check the default ingress certificate validity:

$ oc get secret router-certs-default  -oyaml -n openshift-ingress | grep crt | awk '{print $2}' | base64 -d | openssl x509 -noout -dates

7 . Service-signer certificates:

  • Service serving certificates are signed by the service-ca and has a validity of 2 years by default.
  • service-ca validity can be checked by:
$ oc get secrets/signing-key -n openshift-service-ca -o template='{{index .data "tls.crt"}}' | base64 -d | openssl x509 -noout -dates
  • To check the expiry date of all service-signer certs:
$ oc get secrets --all-namespaces -o custom-columns=SERVICENAME:.metadata.name,NAMESPACE:.metadata.namespace,EXPIRY:.metadata.annotations."service\.beta\.openshift\.io/expiry" | grep -v "<none>"

8 . Logging and Monitoring stack certificates:

$ oc get secret -A -o json | jq -r ' .items[] | select( .metadata.annotations."auth.openshift.io/certificate-not-after" | .!=null and fromdateiso8601<='$( date --date='+1year' +%s )' ) | "expiration: \( .metadata.annotations."auth.openshift.io/certificate-not-after" ) \( .type ) -n \( .metadata.namespace ) \( .metadata.name )" ' | sort | column -t
  • To printout all the expiry of the tls secrets:
$ echo -e "NAMESPACE\tNAME\tEXPIRY" && oc get secrets --all-namespaces -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | while read namespace name cert; do echo -en "$namespace\t$name\t"; echo $cert | base64 -d | openssl x509 -noout  -enddate; done | sort | column -t

Notes:

  • All of the above commands are used with -noout -dates, which will only give the certificate validity, and instead -noout -text can also be used to get the complete certificate information.

  • The respective public key files can be retrieved either from the same secrets or the location mentioned inside master nodes.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.