How to Obtain NooBaa DB PostgreSQL Logs and Configurations - OpenShift Data Foundation (ODF) ODF v4.18 and Below

Solution Verified - Updated

Environment

Red Hat OpenShift Data Foundations (RHODF) v4.x
Red Hat OpenShift Container Storage (RHOCS) v4.18 and below

Issue

In the event, it is necessary to review the PostgreSQL (pgsql) logs/configs internally or submit them to an external source such as a BugZilla/JIRA case the following process should assist.

NOTE: In ODF v4.19+, the pgsql logs are native to the pod. No need to go into the pod to investigate, therefore this solution is not needed.

Resolution

WARNING: DO NOT MAKE ANY MODIFICATIONS TO ANY OF THE DIRECTORIES/FILES BELOW WITHOUT THE APPROVAL OF RED HAT SUPPORT, THIS IS SIMPLY THE DATA COLLECTION PROCESS:

If the noobaa-db-pg-0 pod is "Running":

  1. Ensure the pod is running and tarball postgres logs:
$ oc exec -n openshift-storage -it noobaa-db-pg-0 -- tar czvf /tmp/pgsql-logs.tar.gz /var/lib/pgsql/data/userdata/log
  1. Copy the file out of the pod:
$ oc cp noobaa-db-pg-0:/tmp/pgsql-logs.tar.gz ./pgsql-logs-cao_`date +"%Y%m%d"-"%H%M"`.tar.gz

If the noobaa-db-pg-0 pod IS NOT "Running" (e.g. CLBO):

  1. Validate the db-noobaa-db-pg-0 PVC is not full.

    a. Follow the Validate Data Utilization for CephBlockPool/CephFS storageclass in Ceph and Correlate with PVC/Namespace solution.

    b. If the PVC is full, you can expand it using the Expanding the db-noobaa-db-pg PVC - OpenShift Data Foundation (ODF) solution to resolve this issue. Once resolved you can stop here.

    c. If the PVC IS NOT full, proceed to step 2.

  2. Scale down NooBaa Services:

oc -n openshift-storage scale deployment noobaa-operator --replicas 0
oc -n openshift-storage scale deployment noobaa-endpoint --replicas 0
oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 0
  1. Put the noobaa-db-pg-0 pod into sleep.
$ oc -n openshift-storage patch statefulset noobaa-db-pg --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command", "value":["/bin/sh","-c","while true; do sleep 30; done"]}]'
  1. The noobaa-db-pg-0 pod should restart. Once running perform the tarball postgres logs:
$ oc exec -n openshift-storage -it noobaa-db-pg-0 -- tar czvf /tmp/pgsql-logs.tar.gz /var/lib/pgsql/data/userdata/log
  1. Copy the file out of the pod:
$ oc cp noobaa-db-pg-0:/tmp/pgsql-logs.tar.gz ./pgsql-logs-cao_`date +"%Y%m%d"-"%H%M"`.tar.gz
  1. Remove the sleep loop:
$ oc -n openshift-storage patch statefulset noobaa-db-pg --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command", "value":[]}]'
  1. Scale up NooBaa Services:
oc -n openshift-storage scale deployment noobaa-operator --replicas 1
oc -n openshift-storage scale deployment noobaa-endpoint --replicas 1
oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 1

Root Cause

There may be instances where there could be an underlying database problem causing issues such as slow performance. A deeper look at the config/pgsql logs may reveal the root cause or provide further insight to a BugZilla or JIRA case currently being worked by Engineering.

Diagnostic Steps

Sep-7 9:29:04.824 [WebServer/41]    [L0] core.server.node_services.nodes_monitor:: noobaa-internal-agent-607e9947d19dcf00239536c3 not writable. reasons: node offline, storage is full (Thu Jan 01 1970 00:00:00 GMT+0000 (Coordinated Universal Time)) - [message repeated 200 times since Thu Sep 07 2023 09:28:41 GMT+0000 (Coordinated Universal Time)]

sh-4.4$ psql
psql (12.12)
Type "help" for help.

postgres=# \l+
                                                                   List of databases
    Name    |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   |  Size   | Tablespace |                Description
------------+----------+----------+------------+------------+-----------------------+---------+------------+--------------------------------------------
 nbcore     | postgres | UTF8     | C          | en_US.utf8 |                       | 3517 MB | pg_default |
 nbcore_old | noobaa   | UTF8     | en_US.utf8 | en_US.utf8 |                       | 5408 MB | pg_default |
 postgres   | postgres | UTF8     | en_US.utf8 | en_US.utf8 |                       | 7969 kB | pg_default | default administrative connection database
 template0  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +| 7825 kB | pg_default | unmodifiable empty database
            |          |          |            |            | postgres=CTc/postgres |         |            |
 template1  | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +| 7825 kB | pg_default | default template for new databases
            |          |          |            |            | postgres=CTc/postgres |         |            |
(5 rows)
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.