Perform a One-Time Backup of the noobaa-db-pg-0 Database for the Mulitcloud Object Gateway (NooBaa) for ODF v4.18 and Below

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RCOCP) v4.x
  • Red Hat OpenShift Container Storage (OCS) v4.x
  • Red Hat OpenShift Data Foundations (ODF) v4.x
  • Red Hat Quay (RHQ) v3.x

Issue

  • Customer may decide to backup the MCG separately with PV snapshots and their own Applications. This process is regarding a one-time backup of the noobaa-db-pg-0 database.

  • If there is a process to backup the objects as well, it should take place after running this procedure.

  • This solution includes information on how to backup and restore the encryption keys along with the database for NooBaa.

Related Articles:

Creating an Automated OADP Backup for OpenShift Data Foundation (NooBaa-DB)

Perform a One-Time Backup of the Database for the Multicloud Object Gateway (NooBaa) in ODF v4.19+

Resolution

Backup:

Note: Before backing up the database, please consider the database size along with the PVC size. The database will take up space on the PVC and once the dump is generated, that dump will take up additional space in the PVC. See the solutions below to gauge if the db-noobaa-db-pg-0 PVC needs to be expanded before backing up. Lastly, for large databases greater than ~50GiB, it will take some time to dump the database/copy outside of pod, greater than ~1 hour. Please plan accordingly.

Validate Data Utilization for CephBlockPool/CephFS storageclass in Ceph and Correlate with PVC/Namespace

How to Check the Size/Consumption of the PostgreSQL Database in the db-noobaa-db-pg-0 PVC

Expanding the db-noobaa-db-pg-0 PVC

CONSIDER: The optimal way to perform a db backup (dump) is with the NooBaa services scaled down. Then scaled back up once the dump has completed, as explained in the How to Control Multicloud Object Gateway (MCG)/NooBaa IO Through Scaling of Resources - OpenShift Data Foundation (ODF) solution. It is understood that in some instances, it isn't optimal to stop NooBaa IO. If NooBaa IO cannot be stopped due to critical bucket operations, please ensure the dump is validated, as shown in step 4 of the backup.

If it is possible to scale down the NooBaa services, the steps are provided below:

  • Scale Down Prior to Backup:
$ oc -n openshift-storage scale deployment noobaa-operator noobaa-endpoint --replicas 0
$ oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 0
  • Scale Up Once Backup has Successfully Completed:
$ oc -n openshift-storage scale deployment noobaa-operator noobaa-endpoint --replicas 1
$ oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 1

Procedure:

Backup:

  1. Create a directory for backups to be stored:
$ mkdir noobaa-backup && cd noobaa-backup
  1. Backup secrets to a local folder:
$ for i in $(oc get secret -n openshift-storage | grep ^noobaa | grep -i opaque | awk '{print $1}') ; do oc get secret -n openshift-storage $i -o yaml > ${i}.yaml; done
  1. Backup the PostgreSQL database and save it to a local folder:
$ oc exec -n openshift-storage -it noobaa-db-pg-0 -- bash -c 'pg_dumpall -U postgres > /var/lib/pgsql/data/dump.sql'
$ oc cp -n openshift-storage noobaa-db-pg-0:/var/lib/pgsql/data/dump.sql dump.sql
  1. To validate the db dump once captured, execute the following:
tail -n100 <dump-name>.sql
-- PostgreSQL database dump complete <--------------- SHOULD SEEE THIS
--
-- PostgreSQL database cluster dump complete

Restore:

Note: The /var/lib/pgsql/data folder is persistent. If the pod is still running and the database dump was not deleted from the directory, proceed to step 1. However, if the database dump is no longer in the /var/lib/pgsql/data directory inside the noobaa-db-pg-0 pod, you will have to copy it back into the pod by running the following command:

$ oc cp -n openshift-storage /noobaa-backup/dump.sql noobaa-db-pg-0:/var/lib/pgsql/data/

Procedure:

  1. Stop the NooBaa Service before restoring the NooBaa DB. There will be no object service after this point:
$ oc -n openshift-storage scale deployment noobaa-operator --replicas 0
$ oc -n openshift-storage scale deployment noobaa-endpoint --replicas 0
$ oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 0
  1. Verify that all NooBaa components (except NooBaa DB) have terminated:
$ oc get pods -n openshift-storage | grep noobaa
  1. Login to the NooBaa DB pod and restore DB from a local folder:
    Note: The echo $? output should reveal a 0 to reflect a successful restore.
$ oc -n openshift-storage rsh noobaa-db-pg-0
$ psql -U postgres < /$HOME/data/dump.sql
$ echo $?
$ exit
  1. Start the NooBaa service:
$ oc -n openshift-storage scale deployment noobaa-operator --replicas 1
$ oc -n openshift-storage scale deployment noobaa-endpoint --replicas 1
$ oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 1
  1. Restart the NooBaa pods IAW Product Documentation, ODF v4.18 example will be below:

12.1. Restoring the Multicloud Object Gateway

  1. Optional Step: PERFORM ONLY IF NEEDED. If the NooBaa secrets were not modified/deleted, you will not have to accomplish this step. The encryption keys will be compatible with the restored database. However, if there was a scenario where the secrets/encryption keys were accidentally deleted/modified, this step would restore them.

    a. Delete secrets and restore them from the local folder. Please validate the current errors are related to the encryption keys before proceeding.

Syntax:

$ oc delete secret -n <namespace> <secret-name> ; oc create -f <backup-name>.yaml
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.