How to add an additional CA certificate to the trustedCA of the OpenShift Container Platform proxy in OpenShift Container Platform 4.x?
Environment
- OpenShift Container Platform 4.x
Issue
- Need to add an additional CA (Certificate Authority) certificate to an existing trustedCA bundle certificate in OpenShift 4.x proxy.
Resolution
Important Note: The following information has been provided by Red Hat, but is outside the scope of the posted Service Level Agreements and supported procedures. The information is provided as-is and any configuration settings or installed applications made from the information in this article could make the Operating System unsupported by Red Hat Global Support Services. The intent of this article is to provide information to accomplish the system's needs.
-
Printout and verify the existing proxy configuration:
$ oc get proxy cluster -o yaml apiVersion: config.openshift.io/v1 kind: Proxy ... spec: trustedCA: name: custom-ca #<---- Current used one status: {} -
Display the details of the ConfigMap name (custom-ca) displayed in the previous command output from the output :
# oc get cm custom-ca -n openshift-config -o yaml apiVersion: v1 data: ca-bundle.crt: | -----BEGIN CERTIFICATE----- MIIEDTCCAvWgAwIBAgIUbGvo8ZZvgfKMlGFFW2NO+rIqFpYwDQYJKoZIhvcNAQEL BQAwgZUxCzAJBgNVBAYTAkFFMQ4wDAYDVQQIDAVEdWJhaTEOMAwGA1UEBwwFRHVi YWkxEDAOBgNVBAoMB1JlZCBIYXQxETAPBgNVBAsMCFNlY3VyaXR5MSAwHgYDVQQD DBcqLmFwcHMub2NwNC5leGFtcGxlLmNvbTEfMB0GCSqGSIb3DQEJARYQaW5mb0Bl ..... 0mDDy2D5OJdMpMVJ566OvdhBVWASRnkdhPk+EJ7DC7KOGHqYQXGX3M+bT4iTyQjr q1i22bbpC3eNG0gKt6VQ5WVjtNydSiU2WINQi3mJoL1OGCgr/2foGgZfNt/aLUkw u8vZKZBld6HIsBIAYgm6S3GRKG6nW7JGT6P0BdA5wpZHnd0ybM+ELKLwSsVSglGw gX380MAMz6s+Ql8DMTwPT2wPMFqxcOfJe6/sGdESH7RW -----END CERTIFICATE----- ... -
Copy the existing CA certificate(s) results of the command output showing in the previous step, starting from the first
-----BEGIN CERTIFICATE-----until the last (included)-----END CERTIFICATE----- -
Create a new file containing the certificate copied in the previous step:
# cat <<EOF> custom-ca-bundle.pem -----BEGIN CERTIFICATE----- MIIEDTCCAvWgAwIBAgIUbGvo8ZZvgfKMlGFFW2NO+rIqFpYwDQYJKoZIhvcNAQEL BQAwgZUxCzAJBgNVBAYTAkFFMQ4wDAYDVQQIDAVEdWJhaTEOMAwGA1UEBwwFRHVi YWkxEDAOBgNVBAoMB1JlZCBIYXQxETAPBgNVBAsMCFNlY3VyaXR5MSAwHgYDVQQD DBcqLmFwcHMub2NwNC5leGFtcGxlLmNvbTEfMB0GCSqGSIb3DQEJARYQaW5mb0Bl .... 0mDDy2D5OJdMpMVJ566OvdhBVWASRnkdhPk+EJ7DC7KOGHqYQXGX3M+bT4iTyQjr q1i22bbpC3eNG0gKt6VQ5WVjtNydSiU2WINQi3mJoL1OGCgr/2foGgZfNt/aLUkw u8vZKZBld6HIsBIAYgm6S3GRKG6nW7JGT6P0BdA5wpZHnd0ybM+ELKLwSsVSglGw gX380MAMz6s+Ql8DMTwPT2wPMFqxcOfJe6/sGdESH7RW -----END CERTIFICATE----- EOF -
Remove the unwanted spaces from the previous created file:
# sed -i 's/[[:blank:]][[:blank:]]//g' custom-ca-bundle.pem# cat custom-ca-bundle.pem -----BEGIN CERTIFICATE----- MIIEDTCCAvWgAwIBAgIUbGvo8ZZvgfKMlGFFW2NO+rIqFpYwDQYJKoZIhvcNAQEL BQAwgZUxCzAJBgNVBAYTAkFFMQ4wDAYDVQQIDAVEdWJhaTEOMAwGA1UEBwwFRHVi YWkxEDAOBgNVBAoMB1JlZCBIYXQxETAPBgNVBAsMCFNlY3VyaXR5MSAwHgYDVQQD DBcqLmFwcHMub2NwNC5leGFtcGxlLmNvbTEfMB0GCSqGSIb3DQEJARYQaW5mb0Bl ..... 0mDDy2D5OJdMpMVJ566OvdhBVWASRnkdhPk+EJ7DC7KOGHqYQXGX3M+bT4iTyQjr q1i22bbpC3eNG0gKt6VQ5WVjtNydSiU2WINQi3mJoL1OGCgr/2foGgZfNt/aLUkw u8vZKZBld6HIsBIAYgm6S3GRKG6nW7JGT6P0BdA5wpZHnd0ybM+ELKLwSsVSglGw gX380MAMz6s+Ql8DMTwPT2wPMFqxcOfJe6/sGdESH7RW -----END CERTIFICATE----- -
Append the “custom-ca-bundle.pem“ file generated in the previous step, and add the new desired CA certificate (<New_CA_Cert>.pem):
# cat <New_CA_Cert>.pem >> custom-ca-bundle.pemTo verify and display the content of the CA certificate:
# cat custom-ca-bundle.pem -----BEGIN CERTIFICATE----- MIIEDTCCAvWgAwIBAgIUbGvo8ZZvgfKMlGFFW2NO+rIqFpYwDQYJKoZIhvcNAQEL BQAwgZUxCzAJBgNVBAYTAkFFMQ4wDAYDVQQIDAVEdWJhaTEOMAwGA1UEBwwFRHVi YWkxEDAOBgNVBAoMB1JlZCBIYXQxETAPBgNVBAsMCFNlY3VyaXR5MSAwHgYDVQQD DBcqLmFwcHMub2NwNC5leGFtcGxlLmNvbTEfMB0GCSqGSIb3DQEJARYQaW5mb0Bl eGFtcGxlLmNvbTAeFw0yMjA0MjExMTA5MThaFw0yNzA0MjAxMTA5MThaMIGVMQsw .... 0mDDy2D5OJdMpMVJ566OvdhBVWASRnkdhPk+EJ7DC7KOGHqYQXGX3M+bT4iTyQjr q1i22bbpC3eNG0gKt6VQ5WVjtNydSiU2WINQi3mJoL1OGCgr/2foGgZfNt/aLUkw u8vZKZBld6HIsBIAYgm6S3GRKG6nW7JGT6P0BdA5wpZHnd0ybM+ELKLwSsVSglGw gX380MAMz6s+Ql8DMTwPT2wPMFqxcOfJe6/sGdESH7RW -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIEAzCCAuugAwIBAgIJAL74FeRB/WNeMA0GCSqGSIb3DQEBCwUAMIGXMQswCQYD VQQGEwJBRTEOMAwGA1UECAwFRHViYWkxDjAMBgNVBAcMBUR1YmFpMRAwDgYDVQQK DAdSZWQgSGF0MREwDwYDVQQLDAhTZWN1cml0eTEhMB8GA1UEAwwYKi5hcHBzLm9j cDQuZXhhbXBsZTIuY29tMSAwHgYJKoZIhvcNAQkBFhFpbmZvQGV4YW1wbGUyLmNv ... qccpW7RaQ+3AjZqv5rscuvZGqerYcwM0GMheusPBc73ZUTYiuijdVLW50jWOnN4v zJpsWEtumTMuK5cfVpM3AAqvMjLhvFrpPDdVE8IIaI9C82em3tQpE4NyyCaoW/FQ 3lue2xpJumv4kM92N3xwy+ECLosT8EPL7Fm/+xqMxPjkVhStw7/B5R0Tib+nyytc SAvmmJhx+JXLIINR4jO439NcxFAP1Y4= -----END CERTIFICATE----- -
Create a new ConfigMap using the updated CA certificate file “custom-ca-bundle.pem” generated in the previous step:
# oc create configmap custom-ca-bundle --from-file=ca-bundle.crt=/Full/Path/custom-ca-bundle.pem -n openshift-config configmap/custom-ca-bundle created -
To verify the previous step:
-
List the existing the configMaps and showing the content of the new config map:
``` # oc get configmap -n openshift-config NAME DATA AGE admin-acks 0 10d admin-kubeconfig-client-ca 1 10d custom-ca 1 3h46m custom-ca-bundle 1 2m21s <---- The newly created configmap etcd-ca-bundle 1 10d etcd-metric-serving-ca 1 10d etcd-serving-ca 1 10d initial-kube-apiserver-server-ca 1 10d kube-root-ca.crt 1 10d openshift-install-manifests 2 10d openshift-service-ca.crt 1 10d ``` -
Print out the content of the
custom-ca-bundleconfigmap:``` # oc describe configmap custom-ca-bundle -n openshift-config Name: custom-ca-bundle Namespace: openshift-config Labels: <none> Annotations: <none> Data ==== ca-bundle.crt: ---- -----BEGIN CERTIFICATE----- MIIEDTCCAvWgAwIBAgIUbGvo8ZZvgfKMlGFFW2NO+rIqFpYwDQYJKoZIhvcNAQEL BQAwgZUxCzAJBgNVBAYTAkFFMQ4wDAYDVQQIDAVEdWJhaTEOMAwGA1UEBwwFRHVi YWkxEDAOBgNVBAoMB1JlZCBIYXQxETAPBgNVBAsMCFNlY3VyaXR5MSAwHgYDVQQD DBcqLmFwcHMub2NwNC5leGFtcGxlLmNvbTEfMB0GCSqGSIb3DQEJARYQaW5mb0Bl .... 0mDDy2D5OJdMpMVJ566OvdhBVWASRnkdhPk+EJ7DC7KOGHqYQXGX3M+bT4iTyQjr q1i22bbpC3eNG0gKt6VQ5WVjtNydSiU2WINQi3mJoL1OGCgr/2foGgZfNt/aLUkw u8vZKZBld6HIsBIAYgm6S3GRKG6nW7JGT6P0BdA5wpZHnd0ybM+ELKLwSsVSglGw gX380MAMz6s+Ql8DMTwPT2wPMFqxcOfJe6/sGdESH7RW -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- MIIEAzCCAuugAwIBAgIJAL74FeRB/WNeMA0GCSqGSIb3DQEBCwUAMIGXMQswCQYD VQQGEwJBRTEOMAwGA1UECAwFRHViYWkxDjAMBgNVBAcMBUR1YmFpMRAwDgYDVQQK DAdSZWQgSGF0MREwDwYDVQQLDAhTZWN1cml0eTEhMB8GA1UEAwwYKi5hcHBzLm9j cDQuZXhhbXBsZTIuY29tMSAwHgYJKoZIhvcNAQkBFhFpbmZvQGV4YW1wbGUyLmNv .... qccpW7RaQ+3AjZqv5rscuvZGqerYcwM0GMheusPBc73ZUTYiuijdVLW50jWOnN4v zJpsWEtumTMuK5cfVpM3AAqvMjLhvFrpPDdVE8IIaI9C82em3tQpE4NyyCaoW/FQ 3lue2xpJumv4kM92N3xwy+ECLosT8EPL7Fm/+xqMxPjkVhStw7/B5R0Tib+nyytc SAvmmJhx+JXLIINR4jO439NcxFAP1Y4= -----END CERTIFICATE----- Events: <none> ```
-
Update (patch) the proxy configuration using the new created ConfigMap that contains the bundle CA certificates:
NOTE: This step will create new MachineConfig(s) which will require all nodes to be reboot (one by one).# oc patch proxy/cluster --type=merge --patch='{"spec":{"trustedCA":{"name":"custom-ca-bundle"}}}' proxy.config.openshift.io/cluster patchedVerify that the updated proxy configuration is referencing the newly created ConfigMap :
# oc describe proxy cluster -n openshift-config Name: cluster Namespace: Labels: <none> Annotations: <none> API Version: config.openshift.io/v1 Kind: Proxy Metadata: ... Spec: Trusted CA: Name: custom-ca-bundle Status: Events: <none>
Root Cause
- The trustedCA is already defined and pointed to an existing CA certificate.
Diagnostic Steps
-
Before patching the proxy cluster configuration:
# oc get machineconfigpools NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-673c30076dfc451e09c2068f14398279 True <--- False <--- False 3 3 3 0 9d worker rendered-worker-7beda84535a3cab3806f6378fb3dcc12 True <--- False <--- False 5 5 5 0 9d # oc get machineconfigs|grep -E ^'NAME|rendered' NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE rendered-master-673c30076dfc451e09c2068f14398279 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 9d rendered-worker-7beda84535a3cab3806f6378fb3dcc12 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 9d -
After patching the proxy cluster configuration:
# oc get machineconfigpools
NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE
master rendered-master-673c30076dfc451e09c2068f14398279 False <--- True <--- False 3 0 0 0 9d
worker rendered-worker-7beda84535a3cab3806f6378fb3dcc12 False <--- True <--- False 5 0 0 0 9d
# oc get machineconfigs|grep -E ^'NAME|rendered'
NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE
rendered-master-673c30076dfc451e09c2068f14398279 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 9d
rendered-master-bb27a99d57993349bfa02450f2fc8ce1 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 3m44s <--- This is newly created machineconfig
rendered-worker-201b21f5b2f1ecfd9d54652ad5672679 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 3m44s <--- This is newly created machineconfig
rendered-worker-7beda84535a3cab3806f6378fb3dcc12 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 9d
-
After sometimes from patching the proxy cluster configuration:
# oc get machineconfigpools NAME CONFIG UPDATED UPDATING DEGRADED MACHINECOUNT READYMACHINECOUNT UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT AGE master rendered-master-bb27a99d57993349bfa02450f2fc8ce1 True <--- False False <--- 3 3 3 0 9d worker rendered-worker-201b21f5b2f1ecfd9d54652ad5672679 True <--- False False <--- 5 5 5 0 9d # oc get machineconfigs|grep -E ^'NAME|rendered' NAME GENERATEDBYCONTROLLER IGNITIONVERSION AGE rendered-master-673c30076dfc451e09c2068f14398279 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 9d rendered-master-bb27a99d57993349bfa02450f2fc8ce1 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 26m <--- rendered-worker-201b21f5b2f1ecfd9d54652ad5672679 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 26m <--- rendered-worker-7beda84535a3cab3806f6378fb3dcc12 6cf167014583c41e80407eea5a4eda644f420d26 3.2.0 9d
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.