Error "manifest blob unknown" "blob unknown to registry" while pushing images to registry in OpenShift 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Internal Registry
• NFS

Issue

• While pushing images to the OpenShift 4 registry, the error below appears randomly:

  Error: Error copying image to the remote destination: Error writing manifest: Error uploading manifest 7.5.2 to image-registry/hostname/repository: errors:
  manifest blob unknown: blob unknown to registry
  manifest blob unknown: blob unknown to registry
  

• Error writing manifest in OpenShift 4 registry:

  Error uploading manifest latest to `image-registry/hostname/repository` : manifest blob unknown: blob unknown to registry
  

• NFS is used as a storage backend for RHOCP internal registry.

Resolution

Check if shared storage like NFS is being used by the internal registry pod's backend storage. Shared storage like NFS may impact the consistent writes to the registry, which may cause the issues described in the "Issue" section.

Red Hat recommends using the object storage for registry backend. NFS hosted on Red Hat Enterprise Linux system as a storage backend is not recommended. Refer to is NFS supported for OpenShift cluster internal components in Production? for additional information.

In a non-scaled/high-availability (HA) OpenShift Container Platform registry cluster deployment:

• The preferred storage technology is object storage followed by block storage. The storage technology does not have to support RWX access mode.
• More info on recommended configurable storage technology and This page is not included, but the link has been rewritten to point to the nearest parent document.specific application storage recommendations.

Root Cause

Shared storage like NFS may impact the consistent writes to the registry, which may cause the issues described in the "Issue" section.

Diagnostic Steps

• Check the Internal registry pod logs for the blob unknown errors:

  $ oc get pods -n openshift-image-registry
  [...]
  $ oc logs -n openshift-image-registry [image-registry-pod_name]
  [...]
  

• Check if the issue is same when pushing image from a machine within the cluster and from outside the cluster as well.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.