How to trust a self-signed root CA in Red Hat OpenShift Platform 4.x

Solution Verified - Updated

Environment

Red Hat OpenShift Platform 4.x

Issue

How to trust a self-signed root CA in Red Hat OpenShift Platform 4.x

An external registry is using a custom certificate. How can one tell OpenShift Platform 4.x to trust this certificate?

Resolution

Example setup

In the following examples, the registry uses a certificate that was signed with a self-signed CA. Therefore, hosts that trust the CA certificate will also sign the registry's certificate.

  • The private registry can be found at: 10.10.181.198:5000

  • With username / password: root / password

Base64 encoded:

[cloud-user@user-jump-server openshift]$ echo "cm9vdDpwYXNzd29yZA==" | base64 -d
root:password[cloud-user@test-jump-server openshift]$ echo -n "root:password" | base64
cm9vdDpwYXNzd29yZA==
  • A custom image was pushed to the private registry prior to starting the tests:
sudo buildah bud -t 10.10.181.198:5000/custom-fedora:1.0 .
sudo podman push 10.10.181.198:5000/custom-fedora:1.0
  • And from a test server which trusts that CA, the registry catalog can be verified with:
[cloud-user@test-server containers]$ curl -u root:password https://10.10.181.198:5000/v2/_catalog
{"repositories":["custom-fedora"]}

Trusting a CA or self-signed certificate

Certificates can either be injected directly on cluster creation via additionalTrustBundle or can be pushed after cluster creation.

Pushing with the installer during installation

During installation, add the following in install-config.yaml:

additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  MIIFeTCCA2GgAwIBAgIJAMHSlcBu/fJ0MA0GCSqGSIb3DQEBCwUAMFMxCzAJBgNV
(...)
  T9D7kZYBVs5PEvSw9N3+Z3smVnTiIXZvd/I7W+QcRRX9y/24k7rVRiG5GMx8zSYW
  Mxdq9kFQ2T/bvCsAn6npB97BLsiqOk7XNT2WCBvnyhpZ88lKK074wgoa22bSex9F
  4fatD1KEm3q5h0en9Q==
  -----END CERTIFICATE-----

If the registry is password protected as in this example, also push a pull secret for that registry. 

(...)
pullSecret: '{   
    "auths": {
(...)
        "10.10.181.198:5000": {
            "auth": "cm9vdDpwYXNzd29yZA==",
            "email": "user@redhat.com"
        },
(...)
}
}'
(...)

Verification

After cluster installation, this will have generated the following configuration in the cluster.

For the pullSecret:

[cloud-user@user-jump-server openshift-private-registry]$ oc get secret/pull-secret -n openshift-config -o yaml | grep '\.dockerconfigjson' | awk '{print $NF}' | base64 -d | python -m json.tool | grep 10.10.181.198 -B2 -A3
{
    "auths": {
        "10.10.181.198:5000": {
            "auth": "cm9vdDpwYXNzd29yZA==",
            "email": "user@redhat.com"
        },

And for the CA bundle:

[cloud-user@user-jump-server openshift-private-registry]$ oc get configmap -o yaml -n openshift-config user-ca-bundle
apiVersion: v1
data:
  ca-bundle.crt: |
    -----BEGIN CERTIFICATE-----
    MIIFeTCCA2GgAwIBAgIJAMHSlcBu/fJ0MA0GCSqGSIb3DQEBCwUAMFMxCzAJBgNV
    BAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDERMA8GA1UECgwI
 (...)
    T9D7kZYBVs5PEvSw9N3+Z3smVnTiIXZvd/I7W+QcRRX9y/24k7rVRiG5GMx8zSYW
    Mxdq9kFQ2T/bvCsAn6npB97BLsiqOk7XNT2WCBvnyhpZ88lKK074wgoa22bSex9F
    4fatD1KEm3q5h0en9Q==
    -----END CERTIFICATE-----
kind: ConfigMap
metadata:
  creationTimestamp: "2020-02-04T08:36:19Z"
  name: user-ca-bundle
  namespace: openshift-config
  resourceVersion: "897"
  selfLink: /api/v1/namespaces/openshift-config/configmaps/user-ca-bundle
  uid: 0ec0fc10-91de-446b-9a79-3e9b3805e52b

And on all CoreOS master and worker nodes, one will see:

[root@user-osc-w6h5n-master-0 ~]# cat /etc/pki/ca-trust/source/anchors/openshift-config-user-ca-bundle.crt
-----BEGIN CERTIFICATE-----
MIIFeTCCA2GgAwIBAgIJAMHSlcBu/fJ0MA0GCSqGSIb3DQEBCwUAMFMxCzAJBgNV
BAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDERMA8GA1UECgwI
(...)
T9D7kZYBVs5PEvSw9N3+Z3smVnTiIXZvd/I7W+QcRRX9y/24k7rVRiG5GMx8zSYW
Mxdq9kFQ2T/bvCsAn6npB97BLsiqOk7XNT2WCBvnyhpZ88lKK074wgoa22bSex9F
4fatD1KEm3q5h0en9Q==
-----END CERTIFICATE-----

And the trust will have been updated by CoreOS on all workers and masters:

[root@user-osc-w6h5n-master-0 ~]# curl -u root:password https://10.10.181.198:5000/v2/_catalog
{"repositories":["custom-fedora"]}

[core@user-osc-w6h5n-worker-tkjvv ~]$ curl -u root:password https://10.10.181.198:5000/v2/_catalog
{"repositories":["custom-fedora"]}

Create a test deployment with an image from the external registry:

[cloud-user@test-jump-server containers]$ cat <<'EOF'>fedora-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fedora-deployment
  labels:
    app: fedora-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: fedora-deployment
  template:
    metadata:
      labels:
        app: fedora-deployment
    spec:
      containers:
      - name: fedora
        image: 10.10.181.198:5000/custom-fedora:1.0
        command:
          - sleep
          - infinity
        imagePullPolicy: Always
EOF
[cloud-user@test-jump-server containers]$ kubectl apply -f fedora-deployment.yaml

And make sure that all pods could be deployed for this project:

[cloud-user@test-jump-server containers]$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
fedora-deployment-db4f7df7-62w2l   1/1     Running   0          17s
fedora-deployment-db4f7df7-hx4nc   1/1     Running   0          17s
fedora-deployment-db4f7df7-vrtwm   1/1     Running   0          17s

Pushing a CA or certificate after installation with machineconfig

One way to achieve this is by using a machineconfig to push a new certificate into the master and worker nodes. However, using this method, all master and worker nodes will restart sequentially.

Push the pull secret with the following sequence of commands if the registry is password protected.

oc create secret docker-registry \
    --docker-server=10.10.181.198:5000 \
    --docker-username=root \
    --docker-password=password \
    --docker-email=unused \
    private-registry
oc secrets add serviceaccount/default secrets/private-registry --for=pull

This will generate the following configuration in the cluster:

[cloud-user@test-jump-server openshift-private-registry]$ oc get secret/pull-secret -n openshift-config -o yaml | grep '\.dockerconfigjson' | awk '{print $NF}' | base64 -d | python -m json.tool | grep 10.10.181.198 -B2 -A3
{
    "auths": {
        "10.10.181.198:5000": {
            "auth": "cm9vdDpwYXNzd29yZA==",
            "email": "test@redhat.com"
        },
[cloud-user@test-jump-server openshift]$ oc describe serviceaccount/default
Name:                default
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  default-dockercfg-ttpcl
                     private-registry
Mountable secrets:   default-token-qp47z
                     default-dockercfg-ttpcl
Tokens:              default-token-c7gjc
                     default-token-qp47z
Events:              <none>

In order to push the certificate, insert it into file certificate.txt. Then, create a machine configuration for the masters:

cat<<EOF>ca-trust-master.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: ca-trust-master
  labels:
    machineconfiguration.openshift.io/role: master
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,$(cat certificate.txt  | base64 -w 0)
        filesystem: root
        mode: 0644
        path: /etc/pki/ca-trust/source/anchors/examplecorp-ca.crt
EOF

And create a machine configuration for the workers:

cat<<EOF>ca-trust-worker.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: ca-trust-worker
  labels:
    machineconfiguration.openshift.io/role: worker
spec:
  config:
    ignition:
      version: 2.2.0
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,$(cat certificate.txt  | base64 -w 0)
        filesystem: root
        mode: 0644
        path: /etc/pki/ca-trust/source/anchors/examplecorp-ca.crt
EOF

Apply both files:

oc apply -f ca-trust-worker.yaml
oc apply -f ca-trust-master.yaml

Wait for some time until the machine configuration applies the change and restarts all the nodes:

[cloud-user@test-jump-server openshift]$ oc get nodes
NAME                            STATUS                        ROLES    AGE   VERSION
test-osc-9xz88-master-0       Ready                         master   9h    v1.16.2
test-osc-9xz88-master-1       Ready                         master   9h    v1.16.2
test-osc-9xz88-master-2       NotReady,SchedulingDisabled   master   9h    v1.16.2
test-osc-9xz88-worker-4wjvf   Ready,SchedulingDisabled      worker   9h    v1.16.2
test-osc-9xz88-worker-wzgbl   Ready                         worker   9h    v1.16.2
test-osc-9xz88-worker-xvmf9   Ready                         worker   9h    v1.16.2
[cloud-user@test-jump-server openshift]$ oc get nodes
NAME                            STATUS                        ROLES    AGE   VERSION
test-osc-9xz88-master-0       Ready                         master   9h    v1.16.2
test-osc-9xz88-master-1       Ready                         master   9h    v1.16.2
test-osc-9xz88-master-2       Ready                         master   9h    v1.16.2
test-osc-9xz88-worker-4wjvf   NotReady,SchedulingDisabled   worker   9h    v1.16.2
test-osc-9xz88-worker-wzgbl   Ready                         worker   9h    v1.16.2
test-osc-9xz88-worker-xvmf9   Ready                         worker   9h    v1.16.2
[cloud-user@test-jump-server openshift]$ oc get nodes
NAME                            STATUS                        ROLES    AGE   VERSION
test-osc-9xz88-master-0       Ready                         master   9h    v1.16.2
test-osc-9xz88-master-1       Ready                         master   9h    v1.16.2
test-osc-9xz88-master-2       Ready                         master   9h    v1.16.2
test-osc-9xz88-worker-4wjvf   Ready                         worker   9h    v1.16.2
test-osc-9xz88-worker-wzgbl   Ready                         worker   9h    v1.16.2
test-osc-9xz88-worker-xvmf9   Ready                         worker   9h    v1.16.2

Verification

Once all nodes are up and ready, connect to the nodes via SSH and make sure they can run curl against the registry:

[root@test-osc-9xz88-master-0 ~]#  curl -u root:password https://10.10.181.198:5000/v2/_catalog
{"repositories":["custom-fedora"]}
[root@test-osc-9xz88-master-0 ~]# ls /etc/pki/ca-trust/source/anchors/examplecorp-ca.crt
/etc/pki/ca-trust/source/anchors/examplecorp-ca.crt
[root@test-osc-9xz88-master-0 ~]# cat /etc/pki/ca-trust/source/anchors/examplecorp-ca.crt
-----BEGIN CERTIFICATE-----
MIIFeTCCA2GgAwIBAgIJAMHSlcBu/fJ0MA0GCSqGSIb3DQEBCwUAMFMxCzAJBgNV
(...)
Mxdq9kFQ2T/bvCsAn6npB97BLsiqOk7XNT2WCBvnyhpZ88lKK074wgoa22bSex9F
4fatD1KEm3q5h0en9Q==
-----END CERTIFICATE-----
[root@test-osc-9xz88-master-0 ~]# uptime
 10:14:43 up 8 min,  1 user,  load average: 1.50, 1.30, 0.74

Create a test deployment with an image from the external registry:

[cloud-user@test-jump-server containers]$ cat <<'EOF'>fedora-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fedora-deployment
  labels:
    app: fedora-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: fedora-deployment
  template:
    metadata:
      labels:
        app: fedora-deployment
    spec:
      containers:
      - name: fedora
        image: 10.10.181.198:5000/custom-fedora:1.0
        command:
          - sleep
          - infinity
        imagePullPolicy: Always
EOF
[cloud-user@test-jump-server containers]$ kubectl apply -f fedora-deployment.yaml

And make sure that the pods are properly deployed:

[cloud-user@test-jump-server openshift]$ oc get pods
NAME                               READY   STATUS    RESTARTS   AGE
fedora-deployment-db4f7df7-78pv5   1/1     Running   0          25s
fedora-deployment-db4f7df7-kcbqg   1/1     Running   0          25s
fedora-deployment-db4f7df7-scdqj   1/1     Running   0          25

Pushing with imageconfiguration

Push the pull secret with the following sequence of commands:

oc create secret docker-registry \
    --docker-server=10.10.181.198:5000 \
    --docker-username=root \
    --docker-password=password \
    --docker-email=unused \
    private-registry
oc secrets add serviceaccount/default secrets/private-registry --for=pull

This will generate the following configuration in the cluster after deployment, for the pullSecret:

[cloud-user@akaris-jump-server openshift-private-registry]$ oc get secret/pull-secret -n openshift-config -o yaml | grep '\.dockerconfigjson' | awk '{print $NF}' | base64 -d | python -m json.tool | grep 10.10.181.198 -B2 -A3
{
    "auths": {
        "10.10.181.198:5000": {
            "auth": "cm9vdDpwYXNzd29yZA==",
            "email": "akaris@redhat.com"
        },

[cloud-user@akaris-jump-server openshift]$ oc describe serviceaccount/default
Name:                default
Namespace:           default
Labels:              <none>
Annotations:         <none>
Image pull secrets:  default-dockercfg-ttpcl
                     private-registry
Mountable secrets:   default-token-qp47z
                     default-dockercfg-ttpcl
Tokens:              default-token-c7gjc
                     default-token-qp47z
Events:              <none>

Push the CA bundle with the following sequence of commands:

cat<<'EOF'>registry-ca.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: registry-ca
  namespace: openshift-config
data:
  10.10.181.198..5000: |
    -----BEGIN CERTIFICATE-----
    MIIFeTCCA2GgAwIBAgIJAMHSlcBu/fJ0MA0GCSqGSIb3DQEBCwUAMFMxCzAJBgNV
(...)
    Mxdq9kFQ2T/bvCsAn6npB97BLsiqOk7XNT2WCBvnyhpZ88lKK074wgoa22bSex9F
    4fatD1KEm3q5h0en9Q==
    -----END CERTIFICATE-----
EOF
oc apply -f registry-ca.yaml
oc patch image.config.openshift.io/cluster  --type=merge -p '{"spec":{"additionalTrustedCA":{"name":"registry-ca"}}}'

Note that the ":" as a separator between registry IP address and registry port needs to be expressed as "..".

Verification

Deploy the following deployment: 

cat<<'EOF'>fedora-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fedora-deployment
  labels:
    app: fedora-deployment
spec:
  replicas: 3
  selector:
    matchLabels:
      app: fedora-deployment
  template:
    metadata:
      labels:
        app: fedora-deployment
    spec:
      containers:
      - name: fedora
        image: 10.10.181.198:5000/custom-fedora:1.0
        command:
          - sleep
          - infinity
        imagePullPolicy: Always
EOF
kubectl apply -f fedora-deployment.yaml

Verify that all pods come up:

[cloud-user@jump-server openshift]$ oc get pods
NAME                                READY   STATUS    RESTARTS   AGE
fedora-deployment-d7f8bc7cf-ds4cf   1/1     Running   0          3m6s
fedora-deployment-d7f8bc7cf-j8j2m   1/1     Running   0          3m6s
fedora-deployment-d7f8bc7cf-lzp7j   1/1     Running   0          3m6s

Resources:

Product(s)
Components

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.