What to do when encountering "manifest unknown error: unable to retrieve source image " while using oc-mirror?

Solution Verified - Updated

Environment

OCP v4.11+

Issue

While mirroring operator catalogs with oc-mirror plugin, one might encounter the following errors :

$ oc mirror --config=imageset-config.yaml file:///largefs-dir 
 error: unable to retrieve source image registry.redhat.io/compliance/openshift-selinuxd-rhel8 manifest sha256:568d5b06ebeb54770d5edf7407f2e6e34f8e0fa58a412e19a27124ed496b18d5: manifest unknown: manifest unknown
  error: unable to retrieve source image

Resolution

The recommendation is first and foremost to notify the operator’s maintainers and wait for the problem to be fixed. Trying to skip this image or ignore the error might lead to a seemingly correct mirroring, but result in failure to install or upgrade the operator on the cluster due to the absence of the image in the mirror registry.

In case the user needs the mirroring to pass while the incident is being resolved, the user can choose one of the following approaches:

  1. Option 1: skip the mirroring of the operator referencing the inexistent digest, by removing that operator from the list of operators to mirror from that catalog.
    The user can attempt to add it again and mirror after the problem is resolved.

  2. Option 2: To be applied ONLY if the operator maintainers team confirms that there is no harm in skipping the old image:
    Configure oc-mirror plugin to ignore the inexistent image by using blockedImages in the imageSetConfig, like below:

apiVersion: mirror.openshift.io/v1alpha2
kind: ImageSetConfiguration
storageConfig: # <= Use your favorite flavor, example below
  registry:
    imageURL: localhost:5000/test:latest # Stores metadata in an local/remote image 
    skipTLS: true # Disable TLS certificate checking or use plain HTTP
mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-community-index:v4.12  
      # insert here any filtering to select subset of operators
  blockedImages: # Images to block by name or regular expression
    - name: registry.redhat.io/compliance/openshift-selinuxd-rhel8@sha256:568d5b06ebeb54770d5edf7407f2e6e34f8e0fa58a412e19a27124ed496b18d5

oc-mirror will skip mirroring the images listed in blockedImages, and resume the mirroring process.
Using blockedImages should be preferred over using the --continue-on-error flag in the command line.

Root Cause

This can happen when operator maintainers delete an image digest from the registry while the operator catalog still refers to that image digest.

This problem is usually temporary, and should be resolved once the operator maintainers team is notified and refreshes the references to the images needed by the operator in the catalog.

Diagnostic Steps

Before applying this workaround, first make sure that the image is really absent by running the following command:

$ skopeo inspect docker://registry.redhat.io/compliance/openshift-selinuxd-rhel8@sha256:568d5b06ebeb54770d5edf7407f2e6e34f8e0fa58a412e19a27124ed496b18d5
FATA[0000] Error parsing image name "docker://registry.redhat.io/compliance/openshift-selinuxd-rhel8@sha256:568d5b06ebeb54770d5edf7407f2e6e34f8e0fa58a412e19a27124ed496b18d5": reading manifest sha256:568d5b06ebeb54770d5edf7407f2e6e34f8e0fa58a412e19a27124ed496b18d5 in registry.redhat.io/compliance/openshift-selinuxd-rhel8: unknown: Not Found
Product(s)
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.