Multicloud Object Gateway core pod restarting due to non‑deterministic StorageClass selection

Previously, the Multicloud Object Gateway reconciler selected a StorageClass by listing all StorageClasses and choosing the first match for the Ceph RBD provisioner. Because the Kubernetes List API does not guarantee ordering, this resulted in non‑deterministic and sometimes incorrect StorageClass selection, causing the noobaa-core-0 pod to restart after a fresh deployment.

With this fix, the controller now uses a static, deterministic StorageClass name instead of relying on the unordered list.

As a result, StorageClass selection is consistent, preventing configuration issues and pod restart loops.

(This content is not included.DFBUGS-4763)

Incorrect free‑space calculation for PV Pool

Previously, the PV Pool free‑space calculation was performed per pod instead of averaging usage across all nodes. As a result, the Multicloud Object Gateway could report high usage even when only a single pod exceeded 80%.

With this fix, free space is now calculated as the average used space across all pods in the PV Pool.

(This content is not included.DFBUGS-4152)

Multicloud Object Gateway database (NooBaa DB) high availability failure during upgrade

Previously, the MCG database (NooBaa DB) did not have a default value defined for the PostgreSQL shared_buffers parameter. As a result, if no value was provided, the database was configured with shared_buffers=0, causing high availability (HA) to fail during upgrade.

With this fix, a default value of shared_buffers=1G is applied to ensure proper database behavior and prevent HA failures.

(This content is not included.DFBUGS-4746)

Database corruption caused by concurrent PostgreSQL instances during DB pod replacement

Previously, when the database pod was force‑deleted, a new pod started before the old pod’s container had fully stopped. This race condition resulted in two PostgreSQL instances running concurrently on the same data directory and caused database corruption. The HA controller (HAC) in the Multicloud Object Gateway operator was identified as the component triggering the force deletion. In addition, the database PVC used the ReadWriteOnce access mode, which did not prevent concurrent mounts by two containers on the same node.

This fix disables the HAC by default. For new deployments, the database PVC now uses the ReadWriteOncePod access mode to prevent concurrent mounting.

As a result, the DB pod is no longer force‑deleted by internal components, and new deployments benefit from stronger volume protection to prevent corruption even if the pod is force‑deleted manually.

(This content is not included.DFBUGS-4757)

PVCs in PV Pool backingstores inherited irrelevant CPU and memory limits

Previously, when provisioning a PVC for a PV Pool backingstore, the provisioning logic copied all fields into the PVC template. This caused the PVC to incorrectly inherit CPU and memory limits.

With this fix, only the relevant fields are copied into the PVC template, preventing unintended resource settings.

(This content is not included.DFBUGS-3986)

ThanosRuleHighRuleEvaluationWarnings firing due to incorrect Multicloud Object Gateway metric naming

Previously, some Multicloud Object Gateway metrics did not end with the required _total, _sum, _count, or _bucket suffix. As a result, the ThanosRuleHighRuleEvaluationWarnings info alert continued to fire in the Red Hat OpenShift Container Platform web console.

With this fix, the affected metrics now use the appropriate suffixes, preventing this alert from firing for this issue.

(This content is not included.DFBUGS-3822)

Stuck PVs after final sync due to Retain reclaim policy

Previously, after the final sync, temporary PVs/PVCs were deleted, but some PVs remained because their persistentVolumeReclaimPolicy was set to Retain, preventing cleanup.

With this fix, Ramen now properly resolves conflicts during resource updates, ensuring that cleanup is not skipped.

As a result, no PVs remain stuck after failover or final sync.

(This content is not included.DFBUGS-4535)

Certificate rotation race condition caused empty certificate files

Previously, Kubernetes nodes could fail to start after a reboot because certificate files were empty, even though the files and symlinks existed. This occurred due to a race condition during kubelet certificate rotation.

The issue happened because when new certificates were written to disk, the data was not explicitly forced to persist to the physical disk. If a reboot occurred before the OS flushed the buffered data, the certificate files ended up empty.

With this fix, certificate data is now immediately and explicitly written to disk after being generated.

As a result, certificate files remain valid and non‑empty even if certificate rotation occurs during node reboot.

(This content is not included.DFBUGS-3636)

Incorrect MAX AVAIL value shown in ceph df for stretch‑mode clusters

Previously, Red Hat Ceph Storage clusters operating in stretch mode displayed an incorrect MAX AVAIL value in the output of the ceph df command. This resulted in inaccurate reporting of available storage capacity.

With this update, OpenShift Data Foundation now correctly computes and reports the MAX AVAIL metric, ensuring accurate capacity visualization across Ceph pools in stretch‑mode deployments.

This fix improves cluster observability and prevents misinterpretation of storage utilization.

(This content is not included.DFBUGS-1748)

OSD crashes caused by BlueStore Elastic Shared Blob extent‑resharding logic

Previously, a bug in BlueStore’s Elastic Shared Blob extent‑resharding logic caused incorrect allocation‑unit (AU) boundary calculations. This triggered the assertion ceph_assert(diff ⇐ bytes_per_au[pos]) during resharding, and resulted in OSD crashes. The issue was fixed upstream and included in the Ceph 8.1z5 downstream branch.

With this fix, OSDs now handle BlueStore resharding and object deletion operations without crashing.

As a result, OSDs no longer enter crash loops, and placement groups (PGs) avoid degraded or incomplete states caused by this issue. This bug affected OSDs created with bluestore_elastic_shared_blobs=1 in Squid (19.2.x / Ceph 8.x).

(This content is not included.DFBUGS-4828)

Improved Ceph PG autoscaling by removing default target size ratios from DF‑created pools

Previously, Data Foundation set a default target_size_ratio of 0.49 on the data pools it created. Over time, it was observed that having target size ratios on pools led to poor PG autoscaling and balancing, causing delays in rebalancing across pools.

With this fix, Data Foundation‑created pools no longer use a default target size ratio.

As a result, PG autoscaling is faster and overall pool balancing is improved.

(This content is not included.DFBUGS-2665)

Orphaned CSIAddonsNode resources causing errant sidecar connection attempts

Previously, deleting a worker node left behind stale CSIAddonsNode resources owned by the DaemonSet on that node. These orphaned resources caused the csi-addons-controller-manager pod to make repeated, incorrect connection attempts.

With this fix, the controller manager now watches for and removes stale CSIAddonsNode resources automatically.

As a result, no orphaned CSIAddonsNode resources remain in the cluster.

(This content is not included.DFBUGS-4466)

ocs-metrics-exporter and ocs-provider-server pods stuck in Pending state

Previously, the ocs-metrics-exporter and ocs-provider-server pods did not inherit custom tolerations defined under spec.placement.all in the StorageCluster custom resource, causing the pods to remain in a Pending state.

With this fix, tolerations configured under the pod-specific placement keys are correctly applied. Administrators must configure extra tolerations under spec.placement.metrics-exporter for the ocs-metrics-exporter pod and under spec.placement.api-server for the ocs-provider-server pod. The all placement key remains reserved for rook-ceph resources.

A bug that previously overwrote existing tolerations when adding new ones to metrics-exporter or api-server has also been resolved.

(This content is not included.DFBUGS-5253)

Incorrect pool statistics used for pool‑quota monitoring

Previously, the CephPoolQuotaBytesCriticallyExhausted and CephPoolQuotaBytesNearExhaustion alerts evaluated quota status using incorrect pool statistics, causing the UI to display false warnings.

With this fix, these alerts now use the correct pool‑quota values for evaluation.

(This content is not included.DFBUGS-4611)

Regional-DR is not supported in environments deployed on IBM Power

Regional-DR is not supported in OpenShift Data Foundation environments deployed on IBM Power because ACM 2.15 is not supported on this platform for this release. This impacts both new and upgraded deployments on IBM Power.

(This content is not included.DFBUGS-5369)

CIDR range does not persist in csiaddonsnode object when the respective node is down

When a node is down, the Classless Inter-Domain Routing (CIDR) information disappears from the csiaddonsnode object. This impacts the fencing mechanism when it is required to fence the impacted nodes.

Workaround: Collect the CIDR information immediately after the NetworkFenceClass object is created.

(This content is not included.DFBUGS-2948)

DRPCs protect all persistent volume claims created on the same namespace

The namespaces that host multiple disaster recovery (DR) protected workloads protect all the persistent volume claims (PVCs) within the namespace for each DRPlacementControl resource in the same namespace on the hub cluster that does not specify and isolate PVCs based on the workload using its spec.pvcSelector field.

This results in PVCs that match the DRPlacementControl spec.pvcSelector across multiple workloads. Or, if the selector is missing across all workloads, replication management to potentially manage each PVC multiple times and cause data corruption or invalid operations based on individual DRPlacementControl actions.

Workaround: Label PVCs that belong to a workload uniquely, and use the selected label as the DRPlacementControl spec.pvcSelector to disambiguate which DRPlacementControl protects and manages which subset of PVCs within a namespace. It is not possible to specify the spec.pvcSelector field for the DRPlacementControl using the user interface, hence the DRPlacementControl for such applications must be deleted and created using the command line.

Result: PVCs are no longer managed by multiple DRPlacementControl resources and do not cause any operation and data inconsistencies.

(This content is not included.DFBUGS-1749)

Disabled PeerReady flag prevents changing the action to Failover

The DR controller executes full reconciliation as and when needed. When a cluster becomes inaccessible, the DR controller performs a sanity check. If the workload is already relocated, this sanity check causes the PeerReady flag associated with the workload to be disabled, and the sanity check does not complete due to the cluster being offline. As a result, the disabled PeerReady flag prevents you from changing the action to Failover.

Workaround: Use the command-line interface to change the DR action to Failover despite the disabled PeerReady flag.

(This content is not included.DFBUGS-665)

Information about lastGroupSyncTime is lost after hub recovery for the workloads which are primary on the unavailable managed cluster

Applications that are previously failed over to a managed cluster do not report a lastGroupSyncTime, thereby causing the trigger of the alert VolumeSynchronizationDelay. This is because when the ACM hub and a managed cluster that are part of the DRPolicy are unavailable, a new ACM hub cluster is reconstructed from the backup.

Workaround: If the managed cluster to which the workload was failed over is unavailable, you can still failover to a surviving managed cluster.

(This content is not included.DFBUGS-376)

MCO operator reconciles the veleroNamespaceSecretKeyRef and CACertificates fields

When the OpenShift Data Foundation operator is upgraded, the CACertificates and veleroNamespaceSecretKeyRef fields under s3StoreProfiles in the Ramen config are lost.

Workaround: If the Ramen config has the custom values for the CACertificates and veleroNamespaceSecretKeyRef fields, then set those custom values after the upgrade is performed.

(This content is not included.DFBUGS-440)

For discovered apps with CephFS, sync stop after failover

For CephFS-based workloads, synchronization of discovered applications may stop at some point after a failover or relocation. This can occur with a Permission Denied error reported in the ReplicationSource status.

Workaround:

Remove DR option is not available for discovered apps on the Virtual machines page

The Remove DR option is not available for discovered applications listed on the Virtual machines page.

Workaround:

DR Status is not displayed for discovered apps on the Virtual machines page

DR Status is not displayed for discovered applications listed on the Virtual machines page.

Workaround:

Secondary PVCs are not removed when DR protection is removed for discovered apps

On the secondary cluster, CephFS PVCs linked to a workload are usually managed by the VolumeReplicationGroup (VRG). However, when a workload is discovered using the Discovered Applications feature, the associated CephFS PVCs are not marked as VRG-owned. As a result, when the workload is disabled, these PVCs are not automatically cleaned up and become orphaned.

Workaround: To clean up the orphaned CephFS PVCs after disabling DR protection for a discovered workload, manually delete them using the following command:

$ oc delete pvc <pvc-name> -n <pvc-namespace>

(This content is not included.DFBUGS-2827)

Unable to create new OBCs using Multicloud Object Gateway

When provisioning an NSFS bucket via ObjectBucketClaim (OBC), the default filesystem path is expected to use the bucket name. However, if path is set in OBC.Spec.AdditionalConfig, it should take precedence. This behavior is currently inconsistent, resulting in failures when creating new OBCs.

(This content is not included.DFBUGS-3817)

Poor CephFS performance on stretch clusters

Workloads with many small metadata operations might exhibit poor performance because of the arbitrary placement of metadata server pods (MDS) on multi-site Data Foundation clusters.

(This content is not included.DFBUGS-1753)

OSD pods restart during add capacity

OSD pods restart after performing cluster expansion by adding capacity to the cluster. However, no impact to the cluster is observed apart from pod restarting.

(This content is not included.DFBUGS-1426)

Ceph becomes inaccessible and IO is paused when connection is lost between the two data centers in stretch cluster

When two data centers lose connection with each other but are still connected to the Arbiter node, there is a flaw in the election logic that causes an infinite election among Ceph Monitors. As a result, the Monitors are unable to elect a leader and the Ceph cluster becomes unavailable. Also, IO is paused during the connection loss.

Workaround: Shutdown the monitors of any one data zone by bringing down the zone nodes. Additionally, you can reset the connection scores of surviving Monitor pods.

As a result, Monitors can form a quorum and Ceph becomes available again and IOs resumes.

(This content is not included.DFBUGS-425)

SELinux relabelling issue with a very high number of files

When attaching volumes to pods in Red Hat OpenShift Container Platform, the pods sometimes do not start or take an excessive amount of time to start. This behavior is generic and it is tied to how SELinux relabelling is handled by Kubelet. This issue is observed with any filesystem based volumes having very high file counts. In OpenShift Data Foundation, the issue is seen when using CephFS based volumes with a very high number of files. There are multiple ways to work around this issue. Depending on your business needs you can choose one of the workarounds from the knowledgebase solution https://access.redhat.com/solutions/6221251.

(This content is not included.RFE-3327)

Sync stops after PVC deselection

When a PersistentVolumeClaim (PVC) is added to or removed from a group by modifying its label to match or unmatch the group criteria, sync operations may unexpectedly stop. This occurs due to stale protected PVC entries remaining in the VolumeReplicationGroup (VRG) status.

Workaround: Manually edit the VRG’s status field to remove the stale protected PVC:

$ oc edit vrg <vrg-name> -n <vrg-namespace> --subresource=status

(This content is not included.DFBUGS-4012)

UI shows WaitOnUserCleanUp even when automatic cleanup is enabled

The UI incorrectly displays the WaitOnUserCleanUp status even when automatic cleanup is enabled for VMs. This occurs because the UI relies only on the phase and progression fields of the DRPlacementControl to determine cleanup behavior and does not evaluate the more granular AutoCleanup condition that explicitly indicates automatic cleanup.

Workaround: There is no manual workaround required. This state is transient and clears automatically once the progression field advances to Completed. Manual cleanup should be avoided unless the AutoCleanup condition and its corresponding reason in the DRPlacementControl or VRG status indicate otherwise.

During automatic cleanup, the UI may briefly present a misleading status, which can cause temporary confusion until the cleanup completes.

(This content is not included.DFBUGS-5824)

DRPlacementControl shows ProtectionError even after successful relocation

When a relocation completes, the DRPlacementControl may continue to display a ProtectionError status. This occurs because the Protected condition in the DRPlacementControl status incorrectly reports an Error state, even though the relocation has finished (phase: Relocated, progression: Completed).

Workaround: No direct workaround is available. Wait until retrying the NoClusterDataConflict condition is met.

The DR status in the UI remains in the ProtectionError state until the data conflict is resolved.

(This content is not included.DFBUGS-5823)

UI shows "Unauthorized" error and Blank screen with loading temporarily during OpenShift Data Foundation operator installation

During OpenShift Data Foundation operator installation, sometimes the InstallPlan transiently goes missing which causes the page to show unknown status. This does not happen regularly. As a result, the messages and title go missing for a few seconds.

(This content is not included.DFBUGS-3574)