Failed to pull image with x509: certificate signed by unknown authority error when using mirrored registries in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Disconnected environment
  • Mirrored registry

Issue

  • Using a mirrored registry, there are pods in ImagePullBackOff error, and the images cannot be pulled with error Failed to pull image and x509: certificate signed by unknown authority error errors.

  • The image pull fails with the following message shown in the events:

    Failed to pull image "quay.io/openshift-release-dev/ocp-release@sha256:xxxxxx": rpc error: code = Unknown desc = (Mirrors also failed: custom-image-registry-mirror.example.com:[port]/[namespace]0@sha256:xxxxxx: pinging container registry custom-image-registry-mirror.example.com:[port]: Get "https://custom-image-registry-mirror.example.com:[port]/v2/": x509: certificate signed by unknown authority]): quay.io/openshift-release-dev/ocp-release@sha256:xxxxxx: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 10.0.0.1:53: server misbehaving

Resolution

Refer to the documentation for configuring a new mirror registry in the cluster for adding the required CA certificate to allow pulling images from the registry. For additional information, refer to configuring additional trust stores for image registry access.

  • Please obtain the certificate from the mirror registry using below command:
# openssl s_client -connect QUAY_HOSTNAME:443 -showcerts 2>/dev/null </dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > mirror-registry-cert.crt
  • Create a configmap with the certificate using below command:
$ oc create configmap mirror-registry-ca -n openshift-config --from-file=<registry-url>..<port>=/path/to/mirror-registry-cert.crt
  • Modify the image configuration to include the registry and port using below command:
$ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"mirror-registry-ca"}}}' --type=merge

IMPORTANT: If the image registry is using a custom port, the key in the configmap for the CA needs to be set with .. instead of : as explained in above documents: [mirror_address_host]..[port](like for example custom-image-registry-mirror.example.com..8443).

If the CA is not added to the nodes after following the steps in the documentation, please refer to: A registry CA is not added/renewed into nodes when adding it as additionalTrustedCA to the image configuration in OpenShift 4.

Root Cause

A disconnected environment needs a mirror image registry to be set up. In case the mirror image registry needs to be reconfigured, or additional one needs to be added, it is needed to:

  • Add the mirror registry credentials to the global pull secret.
  • Add the CA-signed mirror registry server certificate to the image.config resource.
  • Create an ImageContentSourcePolicy custom resource (ICSP), which associates the mirror registry with the source registry.

Diagnostic Steps

  • For pods failing with ImagePullBackOff error, check the events of the pod, or for the namespace where it is failing:

    $ oc get pods -A -o wide | grep ImagePullBackOff
[...]
$ oc get events -n [namespace_name]
[...]
94m        Warning   Failed    pod/[pod_name]   Failed to pull image "quay.io/openshift-release-dev/ocp-release@sha256:xxxxxx": rpc error: code = Unknown desc = (Mirrors also failed: custom-image-registry-mirror.example.com:[port]/[namespace]0@sha256:xxxxxx: pinging container registry custom-image-registry-mirror.example.com:[port]: Get "https://custom-image-registry-mirror.example.com:[port]/v2/": x509: certificate signed by unknown authority]): quay.io/openshift-release-dev/ocp-release@sha256:xxxxxx: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 10.0.0.1:53: server misbehaving

  • Check if the same error is shown trying to pull the image from a node following the steps in manual podman pull fails within OCP 4 nodes.

  • Check if the additionalTrustedCA is configured in the image.config resource, and if the referred configmap exists and contains the correct CA certificate:

    $ oc get image.config cluster -o yaml
[...]
  additionalTrustedCA:
    name: [configmap_name]
[...]
$ oc get configmap  [configmap_name] -n openshift-config -o yaml
[...]

  • After creating the configmap and add the additionalTrustedCA to the image.config resource, check that the CA is added to the nodes in a directory inside /etc/docker/certs.d/:

    $ oc debug node/[node_name]
[...]
sh-4.4# chroot /host bash
[root@node_name /]# ls -ltrh /etc/docker/certs.d/
[root@node_name /]# ls -ltrh /etc/docker/certs.d/[mirror_address_host]:[port]/

  • With the CA in the right place in the nodes, check again to manually pull manually pull the images as explained in: manual podman pull fails within OCP 4 nodes.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.