NooBaa PV-Pool Backingstore Frequently Flaps between "Rejected" and "Ready" Phase - OpenShift Data Foundation (ODF)

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RCOCP) v4.x
  • Red Hat OpenShift Container Storage (OCS) v4.x
  • Red Hat OpenShift Data Foundations (ODF) v4.x
  • Red Hat Quay (RHQ) v3.x

Issue

With insufficient resources on the pv-pool and sometimes noobaa-core-o, noobaa-endpoint, and noobaa-db-pg-0 are default/too low, the pods may breach the resource threshold, taking a resource(s) from the NooBaa stack offline and causing NooBaa to enter a temporary Rejected state. To overcome this, additional resources may need to be configured.

  • Objects fail to push to Object Bucket Claim
  • Unable to upload images to a Quay registry
  • pv-pool backingstore flaps between "Rejected" and "Ready" Phase
  • Error message: 'blob update invalid'
WARN[0077] Failed, retrying in 1s ... (1/3). Error: writing blob: uploading layer chunked: blob upload invalid

Resolution

Increasing the resources on the pv-pool should be enough to solve this issue. However, please review the performance tuning guide and make any additional resource increases where necessary based on variables described in the solution.

Internal storagecluster (rook-ceph) users, stop here.

Standalone NooBaa Users ONLY See Below:

NOTE: For standalone NooBaa users, please use the solution below. This is for NooBaa users that didn't create NooBaa when an ODF storagecluster was created, or in the ODF Console GUI wizard. A good way to determine if you should be using the below article is to run the following command and ensure that the storagecluster.yaml is empty/not populated:

$ oc get storagecluster -n openshift-storage -o yaml

If the above output shows nothing/minimal information in the yaml, most likely NooBaa was configured by creating a NooBaa Custom Resource using the Quay documentation. If so, use the solution below.

Root Cause

It is common for NooBaa pv-pools to flap between "Rejected" and "Ready" phases when any decent workload is applied to the bucket, especially larger files such as images. This is because NooBaa is designed to use minimal resources in its default state. This is because not everyone uses NooBaa Object Storage. However, if there is a desire to use NooBaa Object Storage, it should be tuned to and tailored to the workload as described in the articles linked in the resolution.

Diagnostic Steps

$ podman push <omitted>:latest --tls-verify=false
Getting image source signatures
Copying blob bfb53aeef85d done
Copying blob e0348fdb2685 skipped: already exists
WARN[0077] Failed, retrying in 1s ... (1/3). Error: writing blob: uploading layer chunked: blob upload invalid <-----
Getting image source signatures
Copying blob bfb53aeef85d done
Copying blob e0348fdb2685 skipped: already exists
WARN[0091] Failed, retrying in 1s ... (2/3). Error: writing blob: uploading layer chunked: blob upload invalid <-----
Getting image source signatures
Copying blob bfb53aeef85d done
Copying blob e0348fdb2685 skipped: already exists
WARN[0105] Failed, retrying in 1s ... (3/3). Error: writing blob: uploading layer chunked: blob upload invalid <-----
Getting image source signatures
Copying blob bfb53aeef85d done
Copying blob e0348fdb2685 skipped: already exists
Error: writing blob: uploading layer chunked: blob upload invalid <-----

$ oc get pod -n openshift-storage noobaa-default-backing-store-noobaa-pod-6ef5f417 -o yaml
    Limits:
      cpu:     100m <----- NOT ENOUGH RESOURCES
      memory:  400Mi <----- NOT ENOUGH RESOURCES
    Requests:
      cpu:     100m <----- NOT ENOUGH RESOURCES
      memory:  400Mi <----- NOT ENOUGH RESOURCES
<omitted-for-space> 
        exitCode: 137 <-------------------------------------OOMKill Code
        finishedAt: "2025-04-17T19:42:27Z"
        reason: OOMKilled <--------------------------------
        startedAt: "2025-04-01T21:28:20Z"
pv-pool Rejected When Large Files are Uploaded
pv-pool Rejected When Large Files are Uploaded
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.