Backup and Restore for Multicloud Object Gateway database (NooBaa DB)

Solution Verified - Updated

Environment

Red Hat OpenShift Container Storage (OCS) 4.8
Please note: This procedure has been only been tested using:

  • Red Hat OpenShift Container Storage (OCS) 4.8
  • Using a single default backing store without namespace buckets or OBCs.

Issue

  • Customers may decide to backup the Multicloud Object Gateway database (NooBaa DB) separately to the PV snapshots and their own applications.
  • If there is a process to backup the objects as well, it should take place after running this procedure.
  • The following document includes information on how to back up and restore the encryption keys for NooBaa

Resolution

The backup and restore procedures of the NooBaa DB can be done using the following steps.
For testing, upload some data

kubectl port-forward -n openshift-storage service/s3 10443:443 &
NOOBAA_ACCESS_KEY=$(kubectl get secret noobaa-admin -n openshift-storage -o json | jq -r '.data.AWS_ACCESS_KEY_ID|@base64d')
NOOBAA_SECRET_KEY=$(kubectl get secret noobaa-admin -n openshift-storage -o json | jq -r '.data.AWS_SECRET_ACCESS_KEY|@base64d')
alias s3='AWS_ACCESS_KEY_ID=$NOOBAA_ACCESS_KEY AWS_SECRET_ACCESS_KEY=$NOOBAA_SECRET_KEY aws --endpoint https://localhost:10443 --no-verify-ssl s3'
s3 ls

s3 mb s3://test
s3 cp <testfolder> s3://test/ --recursive
s3 ls s3://test

Backup

  1. Backup secrets to a local folder
oc get secret noobaa-root-master-key -o yaml > noobaa-root-master-key.yaml
oc get secret noobaa-admin -o yaml > noobaa-admin.yaml
oc get secret noobaa-db -o yaml > noobaa-db.yaml
oc get secret noobaa-operator -o yaml > noobaa-operator.yaml
oc get secret noobaa-server -o yaml > noobaa-server.yaml
oc get secret noobaa-endpoints -o yaml > noobaa-endpoints.yaml
  1. Backup the PostgreSQL database and save it to a local folder
oc exec -n openshift-storage -it noobaa-db-pg-0 -- pg_dump nbcore -f /tmp/test.db -F custom
oc cp noobaa-db-pg-0:/tmp/test.db ./mcg.bck
  1. For testing, write new data to show a little data loss between backup and restore
s3 mb s3://testloss
s3 cp mcg.bck s3://testloss/  
s3 ls s3://testloss/

Restore

  1. Stop MCG reconciliation:
oc patch -n openshift-storage storagecluster ocs-storagecluster \
    --type merge \
    --patch '{"spec": {"multiCloudGateway": {"reconcileStrategy": "ignore"}}}'
  1. Stop the NooBaa Service before restoring the NooBaa DB. There will be no object service after this point:
oc scale deployment noobaa-operator --replicas=0
oc scale deployment noobaa-endpoint --replicas=0
oc scale sts noobaa-core --replicas=0
  1. Verify that all NooBaa components (except NooBaa DB) have 0 replicas:
oc get deployment noobaa-operator
oc get deployment noobaa-endpoint
oc get sts noobaa-core
  1. Login to the NooBaa DB pod and cleanup potential database clients to nbcore:
oc rsh noobaa-db-pg-0
psql -h 127.0.0.1 -p 5432 -U postgres
SELECT pg_terminate_backend (pid) FROM pg_stat_activity WHERE datname = 'nbcore';
  1. Restore DB from a local folder
oc cp ./mcg.bck noobaa-db-pg-0:test.db
oc exec  -n openshift-storage -it noobaa-db-pg-0 -- pg_restore -d nbcore test.db -c
  1. Delete secrets and restore them from a local folder. Please note that verify that there are no errors before you proceed to the next steps.
oc delete secret noobaa-db ; oc create -f noobaa-db.yaml

oc delete secret noobaa-admin ; oc create -f noobaa-admin.yaml

oc delete secret noobaa-root-master-key ; oc create -f noobaa-root-master-key.yaml

oc delete secret noobaa-operator ; oc create -f noobaa-operator.yaml

oc delete secret noobaa-server ; oc create -f noobaa-server.yaml

oc delete secret noobaa-endpoints ; oc create -f noobaa-endpoints.yaml
  1. Restore MCG reconciliation
oc patch -n openshift-storage storagecluster ocs-storagecluster \
    --type merge \
    --patch '{"spec": {"multiCloudGateway": {"reconcileStrategy": "manage"}}}'
  1. Start the NooBaa service:
oc scale deployment noobaa-operator --replicas=1
oc scale deployment noobaa-endpoint --replicas=1
oc scale sts noobaa-core --replicas=1
  1. Restart the NooBaa DB pod: 
oc delete pods -n openshift-storage noobaa-db-pg-0
  1. Check that the old data exists, but not s3://testloss/
Product(s)
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.