A registry CA is not added/renewed into nodes when adding it as additionalTrustedCA to the image configuration in OpenShift 4

Solution Unverified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Additional registries

Issue

Resolution

First of all it is needed to check that the configmap is correct, and the image.config resource is referencing it correctly.

Another possible cause of the issue is the cluster-image-registry-operator pod not working as expected. In some cases, that pod could be failing when pulling the image from the registry (for example, for disconnected OpenShift 4 clusters). In that case, it will be needed to manually pull the image for that pod, to allow the cluster-image-registry-operator to work properly:

  1. Check the image for the cluster-image-registry-operator pod that is failing, and the node in which that pod is failing:

        $ oc get events -n openshift-image-registry | grep cluster-image-registry-operator
    [...] Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:xxxxxxxx [...]

    $ oc get pods -n openshift-image-registry -o wide | grep cluster-image-registry-operator
    cluster-image-registry-operator-xxxxxxxx-xxxxx   0/1     ImagePullBackOff   0          20d    10.0.0.1     [node_name]       <none>           <none>

  2. Access to the node in which the image is failing, and create a directory in /tmp for the certificate (using oc debug node could fail if the debug image needs to be downloaded for the node):

        $ ssh [node_name]

    $ mkdir -p /tmp/certs
    $ cd /tmp/certs/

  3. Create a ca.crt file in that directory, with the content of the registry's CA:

    $ vi ca.crt
-----BEGIN CERTIFICATE-----
[...]

  4. Try to manually pull the image with podman pull, including the directory where the CA was added (change only the [image_name] in the following command):

    $ sudo -i
# podman pull --authfile /var/lib/kubelet/config.json --cert-dir /tmp/certs/ [image_name]

If the image is correctly downloaded this way, the cluster-image-registry-operator should start working, and the CA configured in the image.config resource be added to all the nodes.

Root Cause

There could be different reasons for the CA not being included in the pods, like wrong configuration of the configmap or the image.config resource, or the cluster-image-registry-operator pod not working as expected.

Diagnostic Steps

  • Check that there are no errors in the status of the image.config resource:

    $ oc get image.config cluster -o yaml

  • Check the status of the cluster-image-registry-operator pod:

    $ oc get pods -n openshift-image-registry
[...]
$ oc get pod [cluster-image-registry-operator_name] -n openshift-image-registry -o yaml

  • Check for any errors in the cluster-image-registry-operator logs:

    $ oc logs -n openshift-image-registry [cluster-image-registry-operator_name]
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.