How to replace the certificate authority for the installer system:admin kubeconfig

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4

Issue

• The kubeconfig containing the system:admin client certificate generated by the installer must be invalidated

Resolution

During the installation a new self-signed Certification Authority is generated, this CA is then used to authenticate client API requests. In order to invalidate the kubeconfig generated by the installer a new CA must be re-deployed.

The original CA used to sign the kubeconfig is stored into the admin-kubeconfig-client-ca configmap under the openshift-config namespace.

The CA can be replaced with the following command (WARNING: please read carefully the "Important Notes" section before applying the following command):

$ oc create configmap admin-kubeconfig-client-ca -n openshift-config --from-file=ca-bundle.crt=new-ca.crt \
  --dry-run -o yaml | oc replace -f -

Important Notes

Replacing the client CA can be a dangerous action and in some cases can lock out the administrators form the cluster. Before replacing the client CA, please make sure to:

• not being authenticated with the client certificate that needs to be invalidated
• make sure to have an alternative way to login into the cluster (eg, using an OAuth-authenticated admin user or using a client certificate signed by an additional client CA)

Safe replacement procedure

The following procedure adds the new CA to the clientCA before replacing the default client CA. In this way is possible to test the authentication and then safely replace the default client CA.

1. Generation of a new self-signed CA, this step is optional: in case a corporate or another CA is already existing can be used

    $ export NAME="custom"
    $ export CA_SUBJ="/OU=openshift/CN=admin-kubeconfig-signer-custom"
   
    # set the CA validity to 10 years
    $ export VALIDITY=3650
   
    # generate the CA private key
    $ openssl genrsa -out ${NAME}-ca.key 4096
   
    # Create the CA certificate
    $ openssl req -x509 -new -nodes -key ${NAME}-ca.key -sha256 -days $VALIDITY -out ${NAME}-ca.crt -subj "${CA_SUBJ}"
   

2. Generation of a new system:admin certificate. The client certificate must have the user into the x.509 subject CN field and the group into the O field.

    $ export USER=system:admin
    $ export GROUP=system:masters
    $ export USER_SUBJ="/O=${GROUP}/CN=${USER}"
   
    # create the user CSR
    $ openssl req -nodes -newkey rsa:2048 -keyout ${USER}.key -subj "${USER_SUBJ}" -out ${USER}.csr
   
    # sign the user CSR and generate the certificate, the certificate must have the `clientAuth` extension
    $ openssl x509 -extfile <(printf "extendedKeyUsage = clientAuth") -req -in ${USER}.csr \
           -CA ${NAME}-ca.crt -CAkey ${NAME}-ca.key -CAcreateserial -out ${USER}.crt -days $VALIDITY -sha256
   

3. In order to have a safe replacement, before removing the default CA the new certificate is added as an additional clientCA

    # create the client-ca ConfigMap"
    $ oc create configmap client-ca-custom -n openshift-config --from-file=ca-bundle.crt=${NAME}-ca.crt
   
    # patch the APIServer
    $ oc patch apiserver cluster --type=merge -p '{"spec": {"clientCA": {"name": "client-ca-custom"}}}'
   

4. Now it is safer to run the command to replace the admin-kubeconfig-client-ca CA, as documented above.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.