Monitoring stack certificates in OpenShift Container Platform 4.6

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4.6

Issue

• How to inspect the certificates used by the monitoring stack components ?
• Which are the secrets responsible for the TLS certificate and key pairs used by the monitoring stack ?
• Is it possible to manually rotate the monitoring stack certificates?

Resolution

• Like OCP 3.x, Openshift 4 monitoring stack is also relying on service serving certificates.
• The service serving certificates secrets are of type kubernetes.io/tls and can be listed from the openshift-monitoring namespace as:

$ oc get secrets -n openshift-monitoring | egrep " kubernetes.io/tls"

• All the above secrets contain a TLS cert-key pair and the TLS certificate, being a service serving certificate, has a default validity of 2 years.
• The secret name is of the format <service_name>-tls. and the service serving certificates are intended for securing the service to service communication.

$ oc get secret <secret_name> -oyaml | grep crt | awk '{print $2}' | base64 -d | openssl x509 -noout -dates -issuer -subject

• The issuer/CA of the TLS certificates is openshift-service-serving-signer which is the service-CA and the subject CN for each certificate are in the form service_name.openshift-monitoring.svc

• The expiry date and the corresponding service name can also be retrieved as:

$ oc get secret <secret_name>  -oyaml | egrep "expiry|service-name"

• All of the above mentioned kubernetes.io/tls secrets(service serving secrets) can be rotated manually by just deleting them. See Manually rotate the generated service certificate

$ oc delete secret <secret_name>

• Apart from the service serving secrets there are 3 secrets that also hold the cert-key pairs used by the monitoring components.

1 . kube-etcd-client-certs
2 . grpc-tls
3 . prometheus-k8s-grpc-tls-xyz
4 . thanos-querier-grpc-tls-xyz

• The secret kube-etcd-client-certs holds the CA and client cert-key pair for ETCD monitoring. Both CA and client cert have a defaultvalidity of 10 years.

$ oc get secret  kube-etcd-client-certs  -ojsonpath='{.data.etcd-client-ca\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject

$ oc get secret  kube-etcd-client-certs  -ojsonpath='{.data.etcd-client\.crt}' | base64 -d | openssl x509 -noout  -dates -issuer -subject

• The remaining 3 secrets with grpc is used for the communication-based grpc protocol which happens in ETCD metric collection and all.

• The secret prometheus-k8s-grpc-tls-xyz holds the openshift-cluster-monitoring CA and server cert-key pair. Both CA and client cert have a defaultvalidity of 2 years.

$  oc get secret prometheus-k8s-grpc-tls-xyz -ojsonpath='{.data.server\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject

$ oc get secret prometheus-k8s-grpc-tls-xyz -ojsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject

• The secret thanos-querier-grpc-tls-xyz holds the openshift-cluster-monitoring CA and client cert-key pair used by the thanos-querier . Both CA and client cert have a defaultvalidity of 2 years.

$ oc get secret thanos-querier-grpc-tls-xyz -ojsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject

$ oc get secret thanos-querier-grpc-tls-xyz -ojsonpath='{.data.client\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject

• The last one grpc-tls contains both server and client cert-key pairs embedded inside the above menioned secrets, which are prometheus-k8s-grpc-tls-xyz and thanos-querier-grpc-tls-xyz used for grpc:

$ oc get secret grpc-tls -oyaml -ojsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject

$ oc get secret grpc-tls -oyaml -ojsonpath='{.data.prometheus-server\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject

$ oc get secret grpc-tls -oyaml -ojsonpath='{.data.thanos-querier-client\.crt}' | base64 -d | openssl x509 -noout -dates -issuer -subject

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.