Which OpenShift certificates do rotate automatically and which do not in Openshift 4.x?

Solution Verified - Updated

Environment

Red Hat OpenShift Container Platform 4.X

Issue

Red Hat OpenShift Container Platform 4.x certificate renew process

Resolution

Major certificates to be renewed :

Below are the most important certificates list in OCP and after that how they work and what should be done.

Kubeconfig

  • Certificate which identifies system:admin user
  • Certificate authority used to verify the identity of the api and ingress certificates

Node certificates

  • /var/lib/kubelet/pki/kubelet-client-current.pem
  • /var/lib/kubelet/pki/kubelet-server-current.pem
  • /etc/kubernetes/static-pod-resources/kube-apiserver-pod-x/secrets/kubelet-client
  • /etc/kubernetes/static-pod-resources/kube-controller-manager-pod-x/secrets/etcd-client
  • /etc/kubernetes/static-pod-resources/kube-scheduler-pod-6/secrets/serving-cert
  • /etc/kubernetes/static-pod-resources/etcd-member/system:etcd-server:etcd-x.xxxx.crt
  • /etc/kubernetes/static-pod-resources/etcd-member/system:etcd-peer:etcd-x.xxxx.crt
  • /etc/kubernetes/static-pod-resources/etcd-member/system:etcd-metric:etcd-x.xxxx.crt

Ingress controllers

  • The certificate which is presented by the router.
  • API - The certificate presented by the API server for external requests.
  • API-INT - The certificate presented by the API server for cluster-internal communication.

General information about Openshift certificates:

OpenShift creates it’s own CA and signs it’s own certs for ingress and API services. In OpenShift 3.x it's possible to provide a custom certificate authority, like 4.x as well. Also it's possible to use custom certificates for Ingress Operators and internal registry
In old 4.x versions, only the certificate authorities known to RHCOS could be used to validate the authenticity of a certificate.

On 4.x, the primary certificates of concern are the node certificates. If they expire, the API may become unavailable. The This page is not included, but the link has been rewritten to point to the nearest parent document.disaster recovery process enables recovery from this situation; however, in 4.x, certificate rotation it's mostly automated and it's becoming more automated in time.

What are the known certificate authorities in RHCOS?

# openssl crl2pkcs7 -nocrl -certfile /etc/ssl/certs/ca-bundle.crt | openssl pkcs7 -print_certs -text -noout

Explaining the rotation process:

The renewal of all internal cert/key pairs and their signers (trusts) are automatic. It's done by creating new signing keys (secret), distributing new ca-bundles that contain old and new trust (configmaps), creating and signing new client or serving cert/key pairs (secrets), and them using the new certificates.

The cluster is rotated once in the first day of usage, so all the machinery is exercised before important workloads land on it. Then expiry is set for approximately 30 days and they're renewed again part way through.

Certificates that do not rotate:

Signing key for the kube-apiserver

  • This exists in all kubeconfig files. Once they're updated, all kubeconfigs become invalid. The certificates are rotated by themselves by the signing key for the admin.kubeconfig, so it's possible to need re-authenticate to update the kubeconfig configurations locally.

Signing key for the kube-apiserver and to kubelet client cert

  • In the Openshift 4.1, this was a kubelet restriction. They've since fixed it, so they're likely to rotate at a slow cadence because it is disruptive to workloads, as the MCO restarts the machines.

Certificates that do rotate:

  • Signing key for aggregator
  • Front-proxyclient cert for aggregator
  • Front-proxyclient cert for kube-apiserver
  • Kubeletserving cert for localhost SNI
  • Kube-apiserverserving cert for service network SNI
  • Kube-apiserverserving cert for external LB SNI
  • Kube-apiseverserving cert for internal LB SNI
  • kube-apiserversigning key for kube-controller-manager
  • Client certclient cert for kube-controller-manager
  • Clientsigning key for kube-scheduler client
  • Certclient cert for kube-scheduler
  • Clientsigning key for CSR signing.

ElasticSearch + Cluster Logging

All certificates of the Logging stack renews automatically by the Operator that installs them. Close to the expiration, they'll be renewed.

Please notice that most of this information is WIP to be introduced in the official documentation. Can be found the initial draft This content is not included.here and Content from github.com is not included.here.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.