ODF upgrade from 4.18 to 4.19 stuck with NooBaa in "Creating" phase and "dbClusterStatus" stuck in "Importing"

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4.x
  • Red Hat OpenShift Data Foundation (ODF) 4.19.x (upgrading from 4.18.x)
  • Multi-Cloud Object Gateway (MCG) / NooBaa

Issue

The ODF 4.19 upgrade migrates the NooBaa database from the existing single PostgreSQL instance to a highly available PostgreSQL cluster managed by CloudNativePG (CNPG). If you are hitting this issue, the following is observed:

  • ODF upgrade completed successfully; however, the StorageCluster CR remains in phase Progressing due to the NooBaa CR failing to reconcile with a message: Waiting on Nooba instance to finish initialization.

  • NooBaa S3 services remained unavailable until manual intervention was taken.

  • The NooBaa CR remains in phase Creating with NooBaa CR "message: cluster creation or import is still in progress, skipping update," and the dbClusterStatus: Importing indefinitely.

  • An s3 outage remains as the noobaa-core and noobaa-endpoint pods fail to scale up due to the upgrade sequence requiring the dbClusterStatus to be Ready, ** WARNING: DO NOT MANUALLY SCALE UP noobaa-core and noobaa-endpoint.**

  • The CNPG Cluster reports Phase: Cluster in healthy state

  • The noobaa-operator logs contain the following warnings and errors repeatedly:

  • BackingStore/NamespaceStore remain in phase Connecting

    level=info msg="cnpg:: the cluster spec was changed but the cluster creation or import is still in progress, skipping update" sys=openshift-storage/noobaa
level=error msg="cnpg:: got error reconciling cluster. error: cluster creation or import is still in progress, skipping update" sys=openshift-storage/noobaa
level=warning msg="⏳ Temporary Error: cluster creation or import is still in progress, skipping update" sys=openshift-storage/noobaa

  • The noobaa-db-pg-0 pod remains alongside noobaa-db-pg-cluster-1 and noobaa-db-pg-cluster-2 pods.

Resolution

Scenario 1 - Before Upgrade:

  • It's best to ensure there will be plenty of space for the incoming noobaa-db-pg-cluster-X PVCs. This process happens during the ODF 4.18 → 4.19 upgrade. Please pre-plan by accomplishing the following:
  1. Validate the usage of the db-noobaa-db-pg-0 PVC:
$ oc exec -n openshift-storage noobaa-db-pg-0 -- df -h /var/lib/pgsql/
  1. To ensure there will be plenty of space available in the new PVCs, patch the storaecluster.yaml prior to upgrading:

Syntax:

$ oc patch -n openshift-storage storagecluster ocs-storagecluster --type merge --patch '{"spec": {"resources": {"noobaa-db-vol":{"requests":{"storage":"<storage-size>"}}}}}'

Example patching 800Gi:

$ oc patch -n openshift-storage storagecluster ocs-storagecluster --type merge --patch '{"spec": {"resources": {"noobaa-db-vol":{"requests":{"storage":"800Gi"}}}}}'
  1. Validate NooBaa resources:
$ oc get storagecluster ocs-storagecluster -n openshift-storage -o json | jq '.spec.resources'
  1. Ensure sufficient maintenance window for large databases and monitor CNPG migration progress during upgrade.

WARNING: It will be important to validate whether the environment is in Scenario 2 or Scenario 3. Most users will be in Scenario 2. However, Scenario 3 may happen when users mistakenly scale up the noobaa-core-0 and the noobaa-endpoint pods manually. When in Scenario 3, the backingstore will become Ready and data is written to the old database. Scenario 3 will have to re-accomplish the DB import or risk data loss.

Scenario 2 - Only the Operator and DB Pods are up (Most Common):

  1. Navigate to the "Diagnostic Steps" in this solution to validate the environment is meeting Scenario 2 criteria.

  2. If the criteria is met, and ONLY the following pods are observed, proceed to Scenario 2, step 3.

$ oc get pods -n openshift-storage -l app=noobaa
NAME                                                              READY   STATUS      RESTARTS   AGE
noobaa-db-pg-0                                                    1/1     Running     0          11h
noobaa-db-pg-cluster-1                                            1/1     Running     0          10h
noobaa-db-pg-cluster-2                                            1/1     Running     0          10h
noobaa-operator-ccc69878f-d8dvd                                   1/1     Running     0          49m
  1. Run the following patch command to reconcile the NooBaa CR and transition the old databse to the new database:
$ oc patch noobaa noobaa -n openshift-storage --type merge --subresource status -p '{"status":{"dbStatus":{"dbClusterStatus":"Ready"}}}'
  1. Scenario 2, step 3 should result in the noobaa-db-pg-0 pod scaling DOWN, with the noobaa-core-0 and noobaa-endpoint pods scaling UP automatically.

Scenario 3 - All NooBaa Pods, Including the Old and New Database Pods are Running (Less Common, but Most Crucial):

  1. Navigate to the "Diagnostic Steps" in this solution to validate the environment is meeting Scenario 3 criteria.

  2. If the criteria is met, this means data has been written to the old database, and not the new database. There will be a disparity between the old/new database. Open a Support Case with ODF Support. The steps are documented in the Private Notes of this solution, and if not followed correctly, could lead to data loss.

Root Cause

During the ODF 4.18 to 4.19 upgrade, the database is migrated from a standalone PostgreSQL container to a CloudNativePG (CNPG) PostgreSQL cluster via pg_dump and pg_restore.
With databases that take longer than a few minutes to import, the new operator (v4.19) starts and reconciles before the old operator transitions the database status out of Importing to Ready.

A race condition in pkg/system/db_reconciler.go causes the operator to skip reconciliation if a difference in the database spec (introduced by the 4.19 upgrade) is detected while the status is still marked as Importing. This creates a deadlock: the status never transitions to Ready because the reconcile loop errors out early, and the reconcile loop continues to fail because the status remains in Importing.

This issue is tracked under Bug Content from redhat.atlassian.net is not included.DFBUGS-8895.

Diagnostic Steps

The expected behvaior is that after the import/join has finished, is that the noobaa-db-pg-0 pod scales down automatically and indefinitely, and the noobaa-db-pg-cluster-1 and noobaa-db-pg-cluster-2 pods take over as primary/secondary DB instances managed by CNPG. Additionally, the noobaa-core-0 and noobaa-endpoint pods scale up automatically, and the NooBaa CR/backingstore(s) become Ready.

Scenario 2 Diagnostic Steps:

  1. Validate that the old DB pod still exists alongside the new DB pods that DO NOT have a -join or -import job pod associated with them. For example, noobaa-db-pg-cluster-1-join or noobaa-db-pg-cluster-1-import. THESE SHOULD BE THE ONLY PODS THAT EXIST IN THE OUTPUT:
$ oc get pod -n openshift-storage -l app=noobaa

noobaa-db-pg-0  <------------ Old DB still remains                1/1     Running     0          1d
noobaa-db-pg-cluster-1   <--- NO JOIN OR IMPORT MESSAGE           1/1     Running     0          10h
noobaa-db-pg-cluster-2   <--- NO JOIN OR IMPORT MESSAGE           1/1     Running     0          10h
noobaa-operator-ccc69878f-d8dvd                                   1/1     Running     0          49m
  1. Validate the NooBaa CR is in the following state:
$ oc get noobaa noobaa -n openshift-storage -o yaml | grep -E 'dbClusterStatus|phase'

    dbClusterStatus: Importing <----
  phase: Creating

$ oc get noobaa noobaa -n openshift-storage -o yaml | grep -A6 lastHeartbeatTime
  - lastHeartbeatTime: "2026-08-05T11:44:20Z"
    lastTransitionTime: "2026-08-05T00:12:41Z"
    message: cluster creation or import is still in progress, skipping update <-------
    reason: TemporaryError
    status: "False"
    type: Available
  1. Validate the Backingstores/Namespacestores are in the following state:
$ oc get backingstore -A

NAME                           TYPE        PHASE        AGE
noobaa-default-backing-store   <omitted>   Connecting   23h


$ oc get namespacestore -A

NAME                             TYPE        PHASE        AGE
noobaa-default-namespace-store   <omitted>   Connecting   23h
  1. Validate the CNPG cluster is reporting healthy:
$ oc get clusters.postgresql.cnpg.noobaa.io noobaa-db-pg-cluster -n openshift-storage -o yaml | grep -i phase

  Phase:            Cluster in healthy state

Scenario 3 Diagnostic Steps (core and endpoint were manually scaled up):

$ oc get pod -n openshift-storage -l app=noobaa

cnpg-controller-manager-7c789f7df9-nx6xk    
noobaa-core-0  <------------------------  SHOULD NOT BE HERE
noobaa-db-pg-0  <------------------------ Old DB still remains
noobaa-db-pg-cluster-1   <--------------- NO JOIN OR IMPORT MESSAGE
noobaa-db-pg-cluster-2   <--------------- NO JOIN OR IMPORT MESSAGE
noobaa-endpoint-56667ddf8c-djbnz   <----- SHOULD NOT BE HERE
noobaa-operator-c58559498-ps9hl
  1. Since core is running, validate the noobaa-core statefulset is still using the old database:
$ oc describe sts -n openshift-storage noobaa-core | grep POSTGRES_HOST
  POSTGRES_HOST:                 noobaa-db-pg-0.noobaa-db-pg.openshift-storage.svc <----- OLD DB HOST
  1. Validate the following in the NooBaa CR:
$ oc get noobaa noobaa -n openshift-storage -o yaml | grep -E 'dbClusterStatus|phase'

    dbClusterStatus: Importing <----
  phase: Creating

$ oc get noobaa noobaa -n openshift-storage -o yaml | grep -A6 lastHeartbeatTime
  - lastHeartbeatTime: "2026-08-05T11:44:20Z"
    lastTransitionTime: "2026-08-05T00:12:41Z"
    message: cluster creation or import is still in progress, skipping update <-------
    reason: TemporaryError
    status: "False"
    type: Available
  1. Validate the Backingstores/Namespacestores are in the following state:
$ oc get backingstore -A

NAME                           TYPE        PHASE        AGE
noobaa-default-backing-store   <omitted>   Connecting   23h


$ oc get namespacestore -A

NAME                             TYPE        PHASE        AGE
noobaa-default-namespace-store   <omitted>   Connecting   23h
  1. Validate the CNPG cluster is reporting healthy:
$ oc get clusters.postgresql.cnpg.noobaa.io noobaa-db-pg-cluster -n openshift-storage -o yaml | grep -i phase

  Phase:            Cluster in healthy state
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.