Supportability for ImageContentSourcePolicy resource to use tags in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat Openshift Container Platform (RHOCP)
    • 4

Issue

  • Can the ImageContentSourcePolicy resource use tags instead of image digests for images?
  • How to use community operators with tags?
  • How to mirror image registries in disconnected environments to refer image tags instead of digests?
  • How to configure mirror-by-digest-only option to allow the ICSP to use tags?
  • How to configure mirror-by-digest-only = false to allow the use of image tags in disconnected environments?
  • How to change pull-from-mirror = "digest-only" in the /etc/containers/registries.conf?

Resolution

Starting from RHOCP 4.13, it is possible to use ImageTagMirrorSet (ITMS) object to pull mirrored images using tags, see This page is not included, but the link has been rewritten to point to the nearest parent document.RHOCP 4.13 release notes.

Workaround before RHOCP 4.13

Important: there is a security risk if using this workaround, as pulling images from mirror registries without the digest specifications could lead to returning different image version from different registry if the image tag mapping is out of sync. Refer to Image SHA Digests vs. Image Tags for more information.

  1. If the mirror was already created in the ImageContentSourcePolicy, remove the one that needs the mirror-by-digest-only = false option from there.

  2. Create a my_registry.conf configuration file, specifying the source and mirror registry and set the mirror-by-digest-only = false option (change the image location accordingly):

     [[registry]]
   prefix = ""
   location = "[original_image_or_registry_namespace_location]"
   mirror-by-digest-only = false

 [[registry.mirror]]
 location = "[mirrored_image_or_registry_namespace_location]"

  3. Encode the file content to the base64 format:

     $ base64 -w0 my_registry.conf

  4. Create a worker-mirror-by-digest-registries.yaml file that can be applied to the MCO to Content from github.com is not included.change the cluster configuration. Copy the base64 text to the source field (changing the [base64_code] in the example), and set the /etc/containers/registries.conf.d to the path field:

     apiVersion: machineconfiguration.openshift.io/v1
 kind: MachineConfig
 metadata:
   labels:
     machineconfiguration.openshift.io/role: worker
   name: 99-worker-mirror-by-digest-registries
 spec:
   config:
     ignition:
       version: 3.1.0
     storage:
       files:
       - contents:
           source: data:text/plain;charset=utf-8;base64,[base64_code]
         filesystem: root
         mode: 420
         path: /etc/containers/registries.conf.d/99-worker-mirror-by-digest-registries.conf

    Important: it's needed to create a separate machineconfig for each node role.

  5. Create the worker-mirror-by-digest-registries.yaml machineconfig:

     $ oc  apply -f worker-mirror-by-digest-registries.yaml

  6. Allow the MCO time to rollout the changes to all the nodes, and check that the file 99-mirror-by-digest-registries.conf is created and it has the desired configuration:

     $ oc debug node/[node_name] -- cat /host/etc/containers/registries.conf.d/99-mirror-by-digest-registries.conf

  7. And check that podman is able to use that configuration:

     $ oc debug node/[node_name] -- chroot /host bash -c 'podman info'
   [...]
   registries:
     [original_image_or_registry_namespace_location]:
       Blocked: false
       Insecure: false
       Location:  [original_image_or_registry_namespace_location]
       MirrorByDigestOnly: false
       Mirrors:
       - Insecure: false
         Location: [mirrored_image_or_registry_namespace_location]
       Prefix: [original_image_or_registry_namespace_location]
   [...]

Root Cause

  • ImageContentSourcePolicy configure the /etc/containers/registries.conf for pulling mirrored images only by using digest.
  • In RHOCP 4.13 new CRDs where introduced for allowing mirroring images using tags.
  • For more information regarding why image SHA ID is used instead of image tags, please refer to Image SHA Digests vs. Image Tags.

Diagnostic Steps

Check the registries.conf configuration in the nodes to see if the mirror registry is already configured by the ImageContentSourcePolicy:

$ oc debug node/[node_name] -- cat /host/etc/containers/registries.conf

Check the configuration used by podman (if there are mirrors configured by the ImageContentSourcePolicy, the MirrorByDigestOnly could be missing or be true):

$ oc debug node/[node_name] -- chroot /host bash -c 'podman info'
[...]
    MirrorByDigestOnly: true
[...]
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.