How to Control Multicloud Object Gateway (MCG)/NooBaa IO Through Scaling of Resources - OpenShift Data Foundation (ODF)

Solution Verified - Updated

Environment

Red Hat OpenShift Container Platform (RHOCP) v4.x
Red Hat OpenShift Data Foundations (RHODF) v4.x

Issue

  • How to scale down and scale up Noobaa services ?

  • In some scenarios, we may want to restart all noobaa pods. This solution will explain why.

  • Due to the architectural changes in ODF v4.19 and above, there are differences between how scaling is accomplished between ODF v4.18 and below and ODF v4.19,+ which will be covered in this solution.

Resolution

To completely stop NooBaa IO, apply these steps in this exact order below:

NOTE: Any read/write IO operations to access noobaa buckets will be blocked, resulting in object service being impacted.

  1. For any version, scale down NooBaa services (copy both lines and paste together):
oc -n openshift-storage scale deployment noobaa-operator noobaa-endpoint --replicas 0
oc -n openshift-storage scale statefulsets.apps noobaa-core --replicas 0

NOTE: The noobaa-db PostgreSQL resource is now isolated. The database cannot receive traffic/requests, or updates due to IO workloads on noobaa buckets or internal operations issued by noobaa-core. This is the optimal time to capture a backup of noobaa-db. See the following related articles:

Perform a One-Time Backup of the noobaa-db-pg-0 Database for the Mulitcloud Object Gateway (NooBaa) for ODF v4.18 and Below
Perform a One-Time Backup of the Database for the Multicloud Object Gateway (NooBaa) in ODF v4.19+

  1. To scale down the noobaa-db pod, adhere to your respective version:

    a. ODF v4.18 or lower:

# oc scale sts -n openshift-storage noobaa-db-pg --replicas=0

b. ODF v4.19 or higher:

NOTE: We cannot stop a single instance. We can hibernate the cluster "noobaa-db-pg-cluster" to stop all noobaa-db pods using the kubctl plugin or using an annotation:

  • Check the status:

    # oc exec -n openshift-storage $(oc get pod -n openshift-storage -l app.kubernetes.io/name=cloudnative-pg -oname) -- kubectl cnpg status noobaa-db-pg-cluster

  • Hibernate the cluster to stop noobaa-db pods, using kubectl plugin from the cnpg-controller-manager pod. For more information review the Content from cloudnative-pg.io is not included.Declarative hibernation documentation.

    # oc exec -n openshift-storage $(oc get pod -n openshift-storage -l app.kubernetes.io/name=cloudnative-pg -oname) -- kubectl cnpg hibernate on noobaa-db-pg-cluster

    OR

  • From an OC bastion, add an annotation to the cluster resource:

    # oc annotate cluster -n openshift-storage noobaa-db-pg-cluster --overwrite cnpg.io/hibernation=on
  1. To START all noobaa pods, perform this this order:

a. Start noobaa-db pod:

  • In ODF 4.18 or lower:

    # oc scale sts noobaa-db-pg --replicas=1

  • In ODF 4.19 or higher:

    • rehydrate the cluster to start noobaa-db pods, using kubectl plugin from the cnpg-controller-manager pod
    # oc exec -n openshift-storage $(oc get pod -n openshift-storage -l app.kubernetes.io/name=cloudnative-pg -oname) -- kubectl cnpg hibernate off noobaa-db-pg-cluster

    OR

    • From an OC bastion, set the cnpg.io/hibernation annotation to off:
    # oc annotate cluster -n openshift-storage noobaa-db-pg-cluster --overwrite cnpg.io/hibernation=off
    • Check the status:
    # oc exec -n openshift-storage $(oc get pod -n openshift-storage -l app.kubernetes.io/name=cloudnative-pg -oname) -- kubectl cnpg status noobaa-db-pg-cluster

    b. Scale up noobaa core:

    # oc scale sts -n openshift-storage noobaa-core --replicas=1

    c. Scale up noobaa endpoints:

    # oc scale deployment -n openshift-storage noobaa-endpoint --replicas=1
  1. Scale up the noobaa operator:
# oc scale deployment -n openshift-storage noobaa-operator --replicas=1

Diagnostic Steps

  • Example of a healthy status:
% oc exec -n openshift-storage $(oc get pod -n openshift-storage -l app.kubernetes.io/name=cloudnative-pg -oname) -- kubectl cnpg status noobaa-db-pg-cluster
Cluster Summary
Name                 openshift-storage/noobaa-db-pg-cluster
PostgreSQL Image:    registry.redhat.io/rhel9/postgresql-16@sha256:42f385ac3c9b8913426da7c57e70bc6617cd237aaf697c667f6385a8c0b0118b
Primary instance:    noobaa-db-pg-cluster-2
Primary start time:  2025-10-18 07:05:23 +0000 UTC (uptime 321h27m44s)
Status:              Cluster in healthy state 
Instances:           2
Ready instances:     2
Size:                740M

Continuous Backup status
Not configured

Physical backups
Primary instance not found

Streaming Replication status
Primary instance not found

Instances status
Name                    Current LSN  Replication role  Status  QoS        Manager Version  Node
----                    -----------  ----------------  ------  ---        ---------------  ----
noobaa-db-pg-cluster-1  -            -                 -       Forbidden  Guaranteed       -  emea-odf419-5nvxw-worker-0-bsdrc
noobaa-db-pg-cluster-2  -            -                 -       Forbidden  Guaranteed       -  emea-odf419-5nvxw-worker-0-969nx


Error(s) extracting status
--------------------------
failed to get status by proxying to the pod, you might lack permissions to get pods/proxy: unknown (get pods https:noobaa-db-pg-cluster-1:8000)
failed to get status by proxying to the pod, you might lack permissions to get pods/proxy: unknown (get pods https:noobaa-db-pg-cluster-2:8000)
  • expected status when hibernated:
% oc exec -n openshift-storage $(oc get pod -n openshift-storage -l app.kubernetes.io/name=cloudnative-pg -oname) -- kubectl cnpg status noobaa-db-pg-cluster
Cluster Summary
Name                 openshift-storage/noobaa-db-pg-cluster
PostgreSQL Image:    registry.redhat.io/rhel9/postgresql-16@sha256:42f385ac3c9b8913426da7c57e70bc6617cd237aaf697c667f6385a8c0b0118b
Primary instance:    noobaa-db-pg-cluster-2
Primary start time:  2025-10-18 07:05:23 +0000 UTC (uptime 321h31m25s)
Status:              Cluster in healthy state 
Instances:           2
Ready instances:     0
Size:                container not found

Hibernation
Status   Hibernated
Message  Cluster has been hibernated
Time     2025-10-31 16:35:07 +0000 UTC

Continuous Backup status
Not configured

Physical backups
Primary instance not found

Streaming Replication status
Primary instance not found

Instances status
Name  Current LSN  Replication role  Status  QoS  Manager Version  Node
----  -----------  ----------------  ------  ---  ---------------  ----

%
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.