How to increase the frequency of deleting metadata in the ODF Multicloud Gateway MCG Noobaa DB database when the Noobaa DB is constantly growing

Environment

• OpenShift Data Foundation (ODF) 4.16+

Issue

• By default, the MCG removes deleted records from the Noobaa DB every two hours. For MCG workloads that are constantly high, this can result in a constantly growing Noobaa database.

Resolution

• The db_cleaner process, which removes deleted records from the Noobaa DB, can be configured to run more frequently. How frequently the db_cleaner process can be configured by setting the CONFIG_JS_DB_CLEANER_CYCLE environment variable in the core container of the noobaa-core StatefulSet. This value is in milliseconds and is set to two hours by default: 2 * 60 * 60 * 1000. Reducing it to run every hour instead can be done by setting CONFIG_JS_DB_CLEANER_CYCLE to 3600000 (60 * 60 * 1000):

  $ oc set env statefulset/noobaa-core -n openshift-storage -c core CONFIG_JS_DB_CLEANER_CYCLE=3600000
  statefulset.apps/noobaa-core updated
  

  $ oc get po noobaa-core-0 -o yaml | yq '.spec.containers[0].env[] | select(.name=="CONFIG_JS_DB_CLEANER_CYCLE")'
  name: CONFIG_JS_DB_CLEANER_CYCLE
  value: "3600000"
  

• db_cleaner process can also be configured to delete more records at a time. By default, it deletes 1000 records at a time. This can be configured with the CONFIG_JS_DB_CLEANER_DOCS_LIMIT environment variable in the core container of the noobaa-core StatefulSet.

  $ oc set env statefulset/noobaa-core -n openshift-storage -c core CONFIG_JS_DB_CLEANER_DOCS_LIMIT=50000
  statefulset.apps/noobaa-core updated
  

  $ oc get po noobaa-core-0 -o yaml | yq '.spec.containers[0].env[] | select(.name=="CONFIG_JS_DB_CLEANER_DOCS_LIMIT")'
  name: CONFIG_JS_DB_CLEANER_DOCS_LIMIT
  value: "50000"
  

Root Cause

• By default, the MCG removes deleted records from the Noobaa DB every two hours. For MCG workloads that are constantly high, this can result in a constantly growing Noobaa database.

Diagnostic Steps

• The total number of records in the objectmds table as well as the number of records marked for deletion can be determined with the following two SQL commands, which are run against the Noobaa DB nbcore. If the number of records marked for deletion constantly grows over time and the ratio of deleted records to total records is high, then increasing the frequency of when the db_cleaner process is run and increasing the number of records deleted at a time can help.

  $ oc exec -n openshift-storage -it noobaa-db-pg-0 -- bash -c 'psql'
  psql (12.22)
  Type "help" for help.
  
  postgres=# \c nbcore
  You are now connected to database "nbcore" as user "postgres".
  postgres=# select count(_id) from objectmds;
  postgres=# select count(_id) from objectmds where data ->> 'deleted' like '%_%';
  

• The current frequency of the db_cleaner process can be determined by querying the CONFIG_JS_DB_CLEANER_CYCLE environment variable in the core container of the noobaa-core StatefulSet.

  $ oc get po noobaa-core-0 -o yaml | yq '.spec.containers[0].env[] | select(.name=="CONFIG_JS_DB_CLEANER_CYCLE")'
  name: CONFIG_JS_DB_CLEANER_CYCLE
  value: "3600000"
  

  In the example above, the `db_cleaner` is configured to run every hour (60 * 60 * 1000).  If the environment     variable is unset, then it will run every two hours by default.
  

• The number of records deleted at a time can be determined by querying the CONFIG_JS_DB_CLEANER_DOCS_LIMIT environment variable in the core container of the noobaa-core StatefulSet.

  $ oc get po noobaa-core-0 -o yaml | yq '.spec.containers[0].env[] | select(.name=="CONFIG_JS_DB_CLEANER_DOCS_LIMIT")'
  name: CONFIG_JS_DB_CLEANER_DOCS_LIMIT
  value: "50000"
  

  In the example above, the `db_cleaner` is configured to delete 50 000 records each time it is run.
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.