service mesh operator mirroring failed

Environment

• OpenShift Container Platform
  • 4.x
• Service mesh operator
  • 2.x

Issue

• Image pull failure using skopeo copy or oc adm catalog mirror for service mesh operator 2.x:

error: unable to retrieve source image registry.redhat.io/openshift-service-mesh/pilot-rhel8 manifest sha256:f7c468b5a35bfce54e53b4d8d00438f33a0861549697d14445eae52d8ead9a68:  unsupported: V2 schema 1 manifest digests are no longer supported for image pulls. Use the equivalent V2 schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332

Resolution

• Use the --continue-on-error option to skip the error and continue running during downloading the V2 schema 1 images (mostly from 1.0.11). Those images are not used for service mesh operator 2.x installation. The generated archive file should include all the necessary images for current service mesh operator. The error could be safely ignored.

Root Cause

• Starting 11th July 2022 Red Hat intends to make some changes to the way the Docker Registry HTTP API at registry.redhat.io and registry.access.redhat.com is handled. See more details from Upcoming changes to container registries (V2 schema 1)
  
• Old service mesh operator images are still using version 2 schema 1 manifest digests but they are not being used for recent version of service mesh operator (starting from 2.0)

Diagnostic Steps

• Use below command to validate the images, for example:

[root@bastion ~]# skopeo inspect --raw docker://registry.redhat.io/openshift-service-mesh/pilot-rhel8@sha256:f7c468b5a35bfce54e53b4d8d00438f33a0861549697d14445eae52d8ead9a68
FATA[0004] Error parsing image name "docker://registry.redhat.io/openshift-service-mesh/pilot-rhel8@sha256:f7c468b5a35bfce54e53b4d8d00438f33a0861549697d14445eae52d8ead9a68": reading manifest sha256:f7c468b5a35bfce54e53b4d8d00438f33a0861549697d14445eae52d8ead9a68 in registry.redhat.io/openshift-service-mesh/pilot-rhel8: unsupported: Not Found, or unsupported. V2 schema 1 manifest digest are no longer supported for image pulls. Use the equivalent schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332

Above means this image does not have V2 schema 2 manifest digest (as it is not being able to be inspected)

• As for comparison, inspect the latest version of the same image (found from registry.redhat.io with latest tag):

[root@bastion ~]# skopeo inspect --raw docker://registry.redhat.io/openshift-service-mesh/pilot-rhel8@sha256:40c9b321a118704e12977456a76716b889498f67a4c8c40ff76db64f45c537c | jq .schemaVersion
2

Above shows the image has V2 schema 2 manifest digest.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.