Standalone MCG Deployed via Quay Documentation Fails ODF v4.19 Upgrade

Updated

ENVIRONMENT
Openshift Container Platform (OCP) 4.18+
Openshift Data Foundation (ODF) 4.18+

ISSUE

  • The ODF upgrade from v4.18 to v4.19 will fail if deployed on a standalone MCG that was created via NooBaa CR yaml by following section Creating Standalone Object Gateway in the Quay documentation. This is a legacy deployment and is no longer supported. The configuration will need to move to a StorageCluster managed NooBaa deployment.

  • The noobaa db will have errors about postgres 16 image while db is on 15.

Incompatible data directory. This container image provides
PostgreSQL '16', but data directory is of
version '15'.

This image supports automatic data directory upgrade from
'15', please carefully consult image documentation
about how to use the '$POSTGRESQL_UPGRADE' startup option.
  • Additionally, the HA for noobaa does not get created and the db is CLBO.
    $ oc get pods | grep noobaa :
noobaa-core-0                                      2/2     Running            0              72m
noobaa-db-pg-0                                     0/1     CrashLoopBackOff   19 (37s ago)   72m
noobaa-endpoint-649cfb9f9-5l5q5                    1/1     Running            0              72m
noobaa-operator-65bd4b5b85-nfkx6                   1/1     Running            0              72m

RESOLUTION:

Prior to ODF v4.18 -> ODF v4.19 Upgrade:

  1. If discovering this article prior to upgrading from ODF v4.18 to v4.19, simply follow the Migrating from a Quay Standalone Multicloud Object Gateway (MCG) to an ODF StorageCluster ODF MCG - NooBaa solution. Once the configuration has moved to a StorageCluster-managed deployment, the upgrade will succeed without issue.

Post ODF ODF v4.19 Upgrade:

  1. If discovering this article after the ODF v4.18 -> ODF v4.19 upgrade has already occurred, and the noobaa-db-pg-0 pod is stuck in CLBO due to the following message: 
Incompatible data directory. This container image provides
PostgreSQL '16', but data directory is of
version '15'.

Scale down the noobaa services:

$ oc -n openshift-storage scale deployment noobaa-operator noobaa-endpoint --replicas 0
$ oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 0
  1. Locate the previous registry.redhat.io/rhel9/postgresql-15@sha256:<image-name>. Some examples to ID this is as follows:

  • If there was a proactive upgrade support case opened for the upgrade, and a must-gather report was submitted to the support case. The image name can be found on the noobaa-dd-pg-0 pod.

  • Search your image-registry (disconnected environments).

  • Run the $ oc get csv -n openshift-storage command. Located under the REPLACES column is the outgoing ODF version. Use the version in the dropdown menu of the This content is not included.Red Hat OpenShift Data Foundation Supportability and Interoperability Checker and associate the rhel9/postgresql-15 in the "Image SHA Values" and insert into the registry.redhat.io/rhel9/postgresql-15@sha256:<image-name> image.

  1. Backup the noobaa-db-pg statefulset:
$ oc get sts -n openshift-storage noobaa-db-pg -o yaml > noobaa-db-pg-sts.yaml
  1. Edit the noobaa-db-pg statefulset and replace any registry.redhat.io/rhel9/postgresql-16@sha256:<image-name> with the previous registry.redhat.io/rhel9/postgresql-15@sha256:<image-name> :
$ oc edit sts -n openshift-storage noobaa-db-pg

image: registry.redhat.io/rhel9/postgresql-16@sha256:<omitted> <---REPLACE w/postgresql-15@sha256:<image-name>
  1. Delete the noobaa-db-pg-0 pod, it should come up/Running:
$ oc delete pod -n openshift-storage noobaa-db-pg-0
  1. Once the noobaa-db-pg-0 pod is running, scale up the NooBaa services. The upgrade should begin, and shortly after, the noobaa-db-pg-0 pod will transition to the noobaa-db-pg-cluster-1 and noobaa-db-pg-cluster-2 High Availability (HA), primary/secondary database after the contents are copied from the db-noobaa-db-pg-0 PVC.
$ oc -n openshift-storage scale deployment noobaa-operator noobaa-endpoint --replicas 1
$ oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 1

DIAGNOSTIC STEPS

  • Check the storagecluster CR. To validate if NooBaa was deployed using the Quay documentation, or created with a NooBaa CR yaml instead of through the ODF UI you will observe an empty storagecluster.yaml as shown below:
$ oc get storagecluster -n openshift-storage -o yaml


apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  • Check the logs of the noobaa-db pod to see if it is running postgres 15.
$ oc logs -n openshift-storage noobaa-db-pg-0

The pod logs below show that the noobaa-db-pg statefulset is on a PostgreSQL 16 image with a PostgreSQL 15 userdata directory. The noobaa-db-pg statefulset never reaches pgsql-16 during the ODF v4.19 upgrade. The noobaa-db is upgraded to pgsql-16 when it transitions to the new cluster HA resource (noobaa-db-pg-cluster-1/noobaa-db-pg-cluster-2), primary and secondary.

Incompatible data directory. This container image provides
PostgreSQL '16', but data directory is of <----------------- 
version '15'. <--------------------------------------------- userdata directory

This image supports automatic data directory upgrade from
'15', please carefully consult image documentation
about how to use the '$POSTGRESQL_UPGRADE' startup option.

$ oc get events -n openshift-storage
5s         Normal    Pulled               pod/noobaa-db-pg-0                                 Container image "registry.redhat.io/rhel9/postgresql-16@sha256:34f12364ca2b2b9251cfc87ad626c58e0c368599da5e3723660819c9d4e69322" already present on machine
26s         Warning   BackOff              pod/noobaa-db-pg-0                                 Back-off restarting failed container db in pod noobaa-db-pg-0_openshift-storage(c205bbe1-eea6-49bc-962b-2fa9566da9a8)
Article Type