oc-mirror plugin fails with catalog is invalid and 401 unauthorized error when it hangs for few minutes in RHOCP 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
    • Disconnected
  • oc-mirror CLI

Issue

  • Mirroring images for a disconnected environment using oc-mirror plugin gets hung and then getting failed.

  • Even though the registry credentials are good as per oc-mirror in a disconnected environment fails with a "HTTP 401 Unauthorized" error, mirroring is getting failed.

  • Usually the error occurs in between the mirroring while it is not able to get an image in a specified time limit.

  • The mirror fails with the following messages:

    The rendered catalog is invalid.

    error: error rendering new refs: render reference "registry.redhat.io/redhat/redhat-operator-index:v4.12": error resolving name : pulling from host registry.redhat.io failed with status code [manifests v4.12]: 401 Unauthorized

Resolution

Red Hat is aware of this issue, and it has been fixed for RHOCP 4.14. via This content is not included.OCPBUGS-20137 by errata RHSA-2023:5006.

If facing similar "401 unauthorized" issues in newer releases, please refer to oc-mirror command is failing to execute with 401 unauthorized error.

Root Cause

When several catalogs are mirrored, oc-mirror will loop catalogs and perform, for each, a rendering (with operator-registry's action.Render) followed by mirrorMappings for the contents of that catalog. Using a single operator-registry for all catalogs means that the second time the registry will be used, it is going to start working with a expired token potentially.

Diagnostic Steps

  • Check the oc-mirror command output and look for the errors as below:

        $ oc mirror --config=./imageset-config.yaml \
      docker://registry.example:5000  --loglevel=debug

    level=debug msg=Unauthorized header=Bearer realm="https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth",service="docker-registry",scope="repository:redhat/redhat-operator-index:pull" host=registry.redhat.io
    level=debug msg=do request host=registry.redhat.io request.header.accept=application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */* request.header.user-agent=opm/alpha request.method=HEAD url=https://registry.redhat.io/v2/redhat/redhat-operator-index/manifests/v4.12
    level=debug msg=fetch response received host=registry.redhat.io response.header.cache-control=max-age=0, no-cache, no-store response.header.connection=keep-alive response.header.content-length=99 response.header.content-type=application/json response.header.date=Tue, 07 Feb 2023 16:23:52 GMT response.header.docker-distribution-api-version=registry/2.0 response.header.expires=Tue, 07 Feb 2023 16:23:52 GMT response.header.pragma=no-cache response.header.registry-proxy-request-id=9f09a6c9-61dd-43cb-abcf-98a1452b60e0 response.header.www-authenticate=Bearer realm="https://registry.redhat.io/auth/realms/rhcc/protocol/redhat-docker-v2/auth",service="docker-registry",scope="repository:redhat/redhat-operator-index:pull" response.status=401 Unauthorized url=https://registry.redhat.io/v2/redhat/redhat-operator-index/manifests/v4.12

    The rendered catalog is invalid.

    Run "oc-mirror list operators --catalog CATALOG-NAME --package PACKAGE-NAME" for more information.

    error: error rendering new refs: render reference "registry.redhat.io/redhat/redhat-operator-index:v4.12": error resolving name : pulling from host registry.redhat.io failed with status code [manifests v4.12]: 401 Unauthorized
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.