Perform a One-Time Backup of the Database for the Multicloud Object Gateway (NooBaa) in ODF v4.19+

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RCOCP) v4.x
  • Red Hat OpenShift Data Foundations (ODF) v4.19+ ONLY

Issue

  • Customer may decide to backup the MCG separately with PV snapshots and their own Applications. This process is regarding a one-time backup of the noobaa-db-pg database.

  • If there is a process to backup the objects as well, it should take place after running this procedure.

  • This solution includes information on how to backup and restore the encryption keys along with the database for NooBaa.

Related Articles:
Perform a One-Time Backup of the noobaa-db-pg-0 Database for the Multicloud Object Gateway (NooBaa) for ODF v4.18 and Below
Creating an Automated OADP Backup for OpenShift Data Foundation (NooBaa-DB)
Expanding the db-noobaa-db-pg-0 PVC

Resolution

Backup

Note: Before backing up the database, please consider the database/PVC size. The database will take up space on your bastion, please ensure there is adequate space available. For large databases sizes greater than ~50GiB, it will take some time to dump the database/copy to the bastion. Sometimes greater than ~1 hour. Please plan accordingly. To validate PVC/db usage, the OCP Console can be used, or review the links below:

Validate Data Utilization for CephBlockPool/CephFS storageclass in Ceph and Correlate with PVC/Namespace

How to Check the Size/Consumption of the PostgreSQL Database in the db-noobaa-db-pg-0 PVC

CONSIDER: The optimal way to perform a db backup (dump) is with the NooBaa services scaled down. Then scaled back up once the dump has completed, as explained in the How to Control Multicloud Object Gateway (MCG)/NooBaa IO Through Scaling of Resources - OpenShift Data Foundation (ODF) solution. It is understood that in some instances, it isn't optimal to stop NooBaa IO. If NooBaa IO cannot be stopped due to critical bucket operations, please ensure the dump is validated, as shown in step 4 of the backup.

If it is possible to scale down the NooBaa services, the steps are provided below:

  • Scale Down Prior to Backup:
oc scale deployment -n openshift-storage cnpg-controller-manager --replicas 0
oc -n openshift-storage scale deployment noobaa-operator noobaa-endpoint --replicas 0
oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 0
  • Scale Up Once Backup has Successfully Completed:
oc scale deployment -n openshift-storage cnpg-controller-manager --replicas 1
oc -n openshift-storage scale deployment noobaa-operator noobaa-endpoint --replicas 1
oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 1

Procedure:

  1. Create a directory for backups to be stored:
$ mkdir noobaa-backup && cd noobaa-backup
  1. Backup secrets to a local folder:
oc get secret noobaa-root-master-key-volume -o yaml > noobaa-root-master-key-volume.yaml
oc get secret noobaa-root-master-key-backend -o yaml > noobaa-root-master-key-backend.yaml
oc get secret noobaa-admin -o yaml > noobaa-admin.yaml
oc get secret noobaa-operator -o yaml > noobaa-operator.yaml
oc get secret noobaa-server -o yaml > noobaa-server.yaml
oc get secret noobaa-endpoints -o yaml > noobaa-endpoints.yaml
  1. Identify the Primary and Secondary instances, then backup the PostgreSQL database.
$ oc get clusters.postgresql.cnpg.noobaa.io -n openshift-storage
NAME                   AGE     INSTANCES   READY   STATUS                     PRIMARY
noobaa-db-pg-cluster   2d      2           2       Cluster in healthy state   noobaa-db-pg-cluster-1 <----

NOTE: To avoid high resource usage on the primary db, you can use the secondary instance to perform the dump. Because of the asynchronous nature of the two, this means the backup of the secondary instance will not be the most up-to-date and will lack any recent data that was not yet replicated from the primary. To backup the most recent data, use the primary instance.
Ensure that there is enough disk space to store the file.

Select one:

a. To backup using the secondary instance, perform the following (low usage, but have pending sync):

$ oc exec -n openshift-storage svc/noobaa-db-pg-cluster-r -c postgres -- pg_dumpall -U postgres > nbcore-secondary.sql

b . To backup using the primary instance, perform the following (high resource usage risk, but up-to-date):

$ oc exec -n openshift-storage svc/noobaa-db-pg-cluster-rw -c postgres -- pg_dumpall -U postgres > nbcore-primary.sql
  1. To validate the db dump once captured, execute the following:
tail -n100 <dump-name>.sql
-- PostgreSQL database dump complete <--------------- SHOULD SEEE THIS
--
-- PostgreSQL database cluster dump complete
  1. Backup the cluster resource noobaa-db-pg-cluster:
$ oc get clusters.postgresql.cnpg.noobaa.io -n openshift-storage noobaa-db-pg-cluster -o yaml > noobaa-db-pg-cluster.yaml

Restore

  1. Stop MCG reconciliation:
$ oc patch StorageCluster -n openshift-storage ocs-storagecluster --type='merge' -p '{"spec":{"multiCloudGateway":{"reconcileStrategy":"ignore"}}}'
  1. Stop the NooBaa Services:
oc scale deployment -n openshift-storage noobaa-operator --replicas=0
oc scale deployment -n openshift-storage noobaa-endpoint --replicas=0
oc scale sts -n openshift-storage noobaa-core --replicas=0
  1. After the pods termination has been validated $ oc get pods -n openshift-storage | grep noobaa, delete the existing cluster.
$ oc delete cluster -n openshift-storage noobaa-db-pg-cluster
  1. After the cluster pods, noobaa-db-pg-cluster-1 and noobaa-db-pg-cluster-2 are terminated, recreate the cluster.

    a. First, review the previously captured noobaa-db-pg-cluster.yaml taken from the backup. Under spec ensure the existing bootstrap object reflects the values below. If not, set it to the following:

  bootstrap:
    initdb:
      database: nbcore
      encoding: UTF8
      localeCType: C
      localeCollate: C
      owner: noobaa

b. Then apply the cluster resource:

$ oc apply -f noobaa-db-pg-cluster.yaml
  1. Confirm the cluster status, and wait for it to be Cluster in healthy state:
$ oc get clusters.postgresql.cnpg.noobaa.io -n openshift-storage noobaa-db-pg-cluster
NAME                   AGE   INSTANCES   READY   STATUS                     PRIMARY
noobaa-db-pg-cluster   53m   2           2       Cluster in healthy state   noobaa-db-pg-cluster-1
  1. Restore DB from a local folder on the client machine to the primary instance:
oc exec -n openshift-storage -i svc/noobaa-db-pg-cluster-rw -c postgres  -- psql -U postgres < [path-to-dump-file]
  1. Validate tables:

    a. exec into an interactive shell:

oc exec -n openshift-storage -it svc/noobaa-db-pg-cluster-rw -c postgres  -- psql -U postgres -d nbcore

b. List tables:

nbcore=# \dt+

c. Validate all of the tables have the following characteristics, then exit the shell:

                                            List of relations
 Schema |         Name         | Type  | Owner  | Persistence | Access method |    Size    | Description 
--------+----------------------+-------+--------+-------------+---------------+------------+-------------
 public | <table-name>         | table | noobaa | permanent   | heap          | xx kB      | 
  1. Only perform this step if the original secrets have been deleted or recreated during a rebuild. Delete secrets and restore them from a local folder.
oc delete secret noobaa-admin ; oc create -f noobaa-admin.yaml
oc delete secret noobaa-root-master-key-volume ; oc create -f noobaa-root-master-key-volume.yaml
oc delete secret noobaa-root-master-key-backend ; oc create -f noobaa-root-master-key-backend.yaml
oc delete secret noobaa-operator ; oc create -f noobaa-operator.yaml
oc delete secret noobaa-server ; oc create -f noobaa-server.yaml
oc delete secret noobaa-endpoints ; oc create -f noobaa-endpoints.yaml
  1. Set the MCG reconcileStrategy back to default:
$ oc patch StorageCluster -n openshift-storage ocs-storagecluster --type='merge' -p '{"spec":{"multiCloudGateway":{"reconcileStrategy":"manage"}}}'
  1. Start the NooBaa operator. This should start the core and endpoints pods as well:
$ oc scale deployment -n openshift-storage noobaa-operator --replicas=1
  1. After ~5 minutes, confirm NooBaa is now in a Ready phase:
$ oc get noobaa -n openshift-storage 
$ oc get backingstore -n openshift-storage 
  1. Only perform this step if NooBaa/noobaa-default-backing-store IS NOT in a Ready phase, restart the Noobaa stack:
oc delete pods <noobaa-operator> -n openshift-storage
oc delete pods <noobaa-core> -n openshift-storage
oc delete pods <noobaa-endpoint> -n openshift-storage
oc delete pods <noobaa-db-cluster-1> -n openshift-storage
oc delete pods <noobaa-db-cluster-2> -n openshift-storage
oc delete pods <rgw-pod> -n openshift-storage <---- only if this pod exists
SBR
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.