How to create a certificate with Subject Alternative Name (SAN) extensions for OpenShift 4 mirror registry

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform [RHOCP]
    • 4.X
  • mirror registry for Red Hat Openshift
    • 3.6.x
    • 3.7.x

Issue

  • Creating a certificate with Subject Alternative Name (SAN) extensions for RHOCP 4 mirror registry.
  • During the upgrade from RHOCP 4.9 to 4.10, the upgrade has stalled and issues are observed with etcd health. The MCP upgrade is stalled and there is one master and one worker node in a Ready,SchedulingDisabled state.
1h57m      Warning  FailedCreatePodSandBox           pod/machine-config-daemon-46bps                 Failed to create pod sandbox: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_machine-config-daemon-46bps_openshift-machine-config-operator_9f07de57-5b99-4846-a584-eb4740dbf975_0": initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcb3cb6b22fe4946336955472cfbfd0d47034b837f3113cbcf219be77ad64cae: (Mirrors also failed: [registry.cluster.example.com:5000/ocp4/openshift4@sha256:fcb3cb6b22fe4946336955472cfbfd0d47034b837f3113cbcf219be77ad64cae: pinging container registry registry.cluster.example.com:5000: Get "https://registry.cluster.example.com:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcb3cb6b22fe4946336955472cfbfd0d47034b837f3113cbcf219be77ad64cae: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 10.0.0.10:53: read udp 10.0.0.20:39303->10.0.0.10:53: i/o timeout

Error can be identified as per following event:

error: unable to push quay.io/openshift-release-dev/ocp-v4.0-art-dev: failed to upload blob sha256:608cfe8580a33a376c92843bd07b2cce574614ac7bff6bddbf03d5f5f44d0b61: Get "https://registry.ocp410.xxx.com:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead

Resolution

  • It is needed to recreate the certificate used by the mirror registry to use the SAN format, and update the cluster chain if needed.
  • If a Certified Authority (CA) is needed, it is possible to create a new CA with the following commands: 
$ openssl genrsa -out CA_key.pem 2048

$ openssl req -x509 \
  -new \
  -nodes \
  -sha256 \
  -days 3650 \             # 10 years may be a long time. Adjust accordingly.
  -key CA_key.pem \
  -out CA_cert.pem \
  -subj "/C=XX/ST=SomeState/L=SomeLocation/O=MyOrganization/CN=SomeCommonName"
  • Create a new certificate signing request with the following steps:
$ openssl genrsa -out ssl.key 2048

$ openssl req -new \
  -key ssl.key \
  -out ssl.csr \
  -subj "/CN=myserver.example.com"        # change with the mirror registry FQDN
  • Create the OpenSSL X509v3 extensions file with Subject Alternative Names:
cat <<EOF > myserver.cnf
authorityKeyIdentifier=keyid,issuer
keyUsage=digitalSignature
extendedKeyUsage=serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = myserver.example.com             # change with the mirror FQDN
DNS.2 = myserver                                     # change with the mirror hostname
EOF
  • Sign the certificate request using the CA and including the above extensions file. Note the parameter -extfile:
openssl x509 -req \
  -in ssl.csr \
  -CA CA_cert.pem \
  -CAkey CA_key.pem \
  -CAcreateserial \
  -out ssl.pem \
  -days 825 \                           # Certificate validity. Adjust accordingly.          
  -sha256 \
  -extfile myserver.cnf
  • Replace the mirror registry certificate and key with the new ones, and restart the service.

Root Cause

"Kubernetes is built using golang 1.17. This version of go removes the ability to use a GODEBUG=x509ignoreCN=0 environment setting to re-enable deprecated legacy behavior of treating the CommonName of X.509 serving certificates as a host name."

Diagnostic Steps

  • In the cluster events the certificate error is visible:
$ oc get events
1h57m      Warning  FailedCreatePodSandBox           pod/machine-config-daemon-46bps                 Failed to create pod sandbox: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_machine-config-daemon-46bps_openshift-machine-config-operator_9f07de57-5b99-4846-a584-eb4740dbf975_0": initializing source docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcb3cb6b22fe4946336955472cfbfd0d47034b837f3113cbcf219be77ad64cae: (Mirrors also failed: [registry.cluster.example.com:5000/ocp4/openshift4@sha256:fcb3cb6b22fe4946336955472cfbfd0d47034b837f3113cbcf219be77ad64cae: pinging container registry registry.cluster.example.com:5000: Get "https://registry.cluster.example.com:5000/v2/": x509: certificate relies on legacy Common Name field, use SANs instead]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fcb3cb6b22fe4946336955472cfbfd0d47034b837f3113cbcf219be77ad64cae: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 10.x.x.x:53: read udp 10.x.x.x:x->10.x.x.x:53: i/o timeout
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.