Noobaa certificate verification for NamespaceStore endpoints

Previously, missing validation of CA bundle when mounting NamespaceStore endpoints caused failures in loading and consuming provided CA bundles. Validation for CA bundles has now been added to ensure proper certificate verification.

(This content is not included.DFBUGS-2712)

Support for AWS region ap-east-2 in Noobaa operator

Previously, the ap-east-2 region was missing from the MCG operator-supported regions list, preventing creation of a default BackingStore when deployed in this region. The missing region has now been added to the supported list.

(This content is not included.DFBUGS-2802)

Noobaa no longer fails to issue deletes to RGW

A configuration change caused delays in deleting large numbers of small objects from the underlying RGW storage. This impacted performance during high-volume delete operations. The issue was resolved by reverting the configuration change, eliminating the delay in deletion from the underlying storage.

(This content is not included.DFBUGS-2916)

ACM console view persistence on hard refresh

Previously, a hard refresh from the ACM console caused the view to revert to the OCP (local-cluster) console. This was because Multicluster Orchestrator console routes were not registered properly for ACM (all clusters) view, which disrupted the expected navigation behavior. The routing logic has now been corrected, and refreshing the browser no longer changes the active view. Users remain in the ACM console as intended.

(This content is not included.DFBUGS-4061)

DR status now visible for VMs

The DR Status was missing on the VM list page, and the Remove disaster recovery option was not available when managing the VMs protected using label selectors. This happened because the UI could not correctly identify the VM’s cluster and its DRPC.

The issue was fixed by reading the VM cluster from the correct field and improving how DRPCs are parsed when label selectors are used. Now, both the DR Status and the Remove disaster recovery options work as expected.

(This content is not included.DFBUGS-4286)

Disabling DR for a CephFS application with consistency groups enabled no longer leaves some resources behind

Disabling DR for a CephFS application with consistency groups enabled no longer leaves any resources behind. Manual cleanup is no longer required.

(This content is not included.DFBUGS-2950)

s3StoreProfile in ramen-hub-operator-config after upgrade from 4.18 to 4.19

Previously, after upgrading from 4.18 to 4.19, the ramen-hub-operator-config ConfigMap was overwritten with default values from the Ramen-hub CSV. This caused loss of custom S3Profiles and other configurations added by the Multicluster Orchestrator (MCO) operator. The issue has been fixed to preserve custom entries during upgrade, preventing disruption in S3 profile configurations.

(This content is not included.DFBUGS-3634)

virtualmachines.kubevirt.io resource no longer fails restore due to mac allocation failure on relocate

Previously, when a virtual machine was relocated back to the preferred cluster, the relocation could fail because its MAC address was unavailable. This occurred if the virtual machine was not fully cleaned up on the preferred cluster after being failed over to the failover cluster. This cleanup process has been corrected, ensuring successful relocation to the preferred cluster.

(This content is not included.BZ#2295404)

Failover process no longer fails when the ReplicationDestination resource has not been created yet

Previously, if the user initiated a failover before the LastGroupSyncTime was updated, the failover process would fail. This failure was accompanied by an error message indicating that the ReplicationDestination does not exist.

This issue has been resolved, and failover works as expected.

(This content is not included.DFBUGS-632)

After Relocation of consistency groups based workload, synchronization no longer stops

Previously, when applications using CephRBD volumes with volume consistency groups were running and the secondary managed cluster went offline, replication for these volumes could stop indefinitely—even after the secondary cluster came back online. During this condition, the Volume SynchronizationDelay alert was triggered, starting with a Warning status and later escalating to Critical, indicating replication had ceased for the affected volumes. This issue has been resolved to ensure replication resumes automatically when the secondary cluster is restored.

(This content is not included.DFBUGS-3812)

Ceph monitor endpoints fully visible

Previously, only one of the three Ceph monitor endpoints was showing up due to missing entries in the CSI ConfigMap. This meant only one of the mons was there for fault tolerance.

The issue was fixed by adding all monitor endpoints to the ConfigMap. Now, all mons are visible, and CSI communication is fault-tolerant.

(This content is not included.DFBUGS-4344)

Fixed StorageSystem creation wizard issues

Previously, the Network Type field for Host was missing, resulting in empty network details and a misleading tooltip that described Multus instead of the actual host configuration. This caused confusion in the summary view, where users saw no network information and an inaccurate tooltip.

With this update, the tooltips were removed and replaced with radio buttons featuring correct labels and descriptions.

(This content is not included.DFBUGS-2582)

Force delete option restored for stuck StorageConsumer

Previously, users were unable to forcefully delete a StorageConsumer resource if it was stuck in a deletion state due to the presence of a deletionTimeStamp.

This issue has been resolved by updating the Actions menu to enable Delete StorageConsumer even when a deletionTimeStamp is present. As a result, you can force delete StorageConsumer resources when required.

(This content is not included.DFBUGS-2819)

Fix for Disaster Recovery misconfiguration after upgrade from v4.17.z to v4.18

Previously, the upgrade process resulted in incorrect DR resource configurations, impacting workloads that rely on ocs-storagecluster-ceph-rbd and ocs-storagecluster-ceph-rbd-virtualization storage classes.

With this fix, the DR resources are correctly configured after the upgrade.

(This content is not included.DFBUGS-1804)

Warning message in the UI right after creation of StorageCluster no longer appears

Previously, a warning popup appeared in the UI during the creation of a StorageSystem or StorageCluster. This was caused by the Virtualization StorageClass not being annotated with storageclass.kubevirt.io/is-default-virt-class: "true", by default, after deployment.

With this fix, the required annotation is applied automatically, preventing unnecessary warnings.

(This content is not included.DFBUGS-2921)

PVC type misclassification resolved in UI

Previously, the UI was incorrectly displaying block PVCs as filesystem PVCs due to outdated filtering method that relied on assumptions based on VRG naming conventions. This led to confusion as the PVC type was inaccurately reported.

To address this, the filter distinguishing block and filesystem PVCs is removed, acknowledging that a group can contain both types. This change eliminates misclassification and ensures accurate representation of PVCs in the UI.

(This content is not included.DFBUGS-4219)

Bucket Lifecycle rule deletion now supported

Previously it was not possible to delete the last remaining bucket lifecycle rule due to a backend error—attempting to update the LifecycleConfiguration with empty rules triggered a 500 response.

This has been fixed by switching to deleteBucketLifecycle for cases where the entire lifecycle configuration needs to be cleaned up. As a result, you can delete all bucket lifecycle rules without encountering errors.

(This content is not included.DFBUGS-2960)

CephFS volume filtering corrected in the UI

Previously, the UI filtering for CephFS volumes was not functioning correctly and also mistakenly excluded the CephFS PVCs when the "block" option was selected. This was due to the outdated filtering method based on VRG naming assumptions that no longer apply.

To resolve this, the block/filesystem filter is removed, recognizing that a group might contain both types of PVCs. This fix eliminates misclassification and ensures accurate display of CephFS volumes in the UI.

(This content is not included.DFBUGS-4065)

Alert for essential OpenShift Data Foundation pods down during capacity addition

Previously, there was no test to check if the essential OpenShift Data Foundation pods were working, leading to an error when adding capacity.

To address this issue, if essential pods are down when attempting to add capacity, the user is alerted and not allowed to proceed.

(This content is not included.DFBUGS-1755)

Support external Red Hat Ceph Storage deployment on KubeVirt nodes

Previously, on OpenShift Container Platform deployed on KubeVirt nodes, there was no option to deploy OpenShift Data Foundation with external Red Hat Ceph Storage (RHCS) due to the Infrastructure CR reporting oVirt and KubeVirt as separate platforms.

With this fix, KubeVirt is added to the allowed list of platforms. As a result, you can create or link external RHCS storage systems from the UI.

(This content is not included.DFBUGS-4018)

Missing Toleration for Prometheus Operator in ROSA HCP Deployments

Previously, it was required to manually patch the pod after creation to apply tolerations.

With this fix, an issue where the prometheus-operator pod in Red Hat OpenShift Service on AWS (ROSA) with hosted control planes (HCP) that was missing the required tolerations was fixed. The tolerations are correctly applied during deployment, eliminating the need for manual intervention.

(This content is not included.DFBUGS-1272)

Service "ocs-provider-server" is invalid: spec.ports[0].nodePort: Invalid value: 31659: provided port is already allocated error no longer appears while reconciling

Previously, the ocs-operator deployed a service using port 31659, which could conflict with an existing nodePort service that is already using the same port. This conflict caused the ocs-operator deployment to fail, resulting in upgrade reconciliation getting stuck.

With this fix, the port allocation is handled more safely to avoid clashes with existing services.

(This content is not included.DFBUGS-1831)

ocs-metrics-exporter inherits node selector

Previously, the ocs-metrics-exporter did not inherit the node selector configuration, causing scheduling issues. This has been resolved by ensuring the node selector is properly applied, as detailed in this Red Hat Solution.

(This content is not included.DFBUGS-3728)

Clone count alert now fires promptly when 200+ clones are created

The clone count alert was previously stuck in a Pending state and failed to fire in a timely manner when over 200 clones were created. This was caused by the alert’s firing threshold being set to 30 minutes, resulting in a long delay. To resolve this, the firing time was reduced from 30 minutes to 30 seconds. As a result, the alert now fires as expected, providing timely notifications when the clone count exceeds the threshold.

(This content is not included.DFBUGS-3869)

Correct runbook URL for HighRBDCloneSnapshotCount alert

The runbook URL linked to the 'HighRBDCloneSnapshotCount' alert was previously incorrect, leading users to a non-existent help page. This issue has been fixed by updating the alert configuration with the correct URL.

(This content is not included.DFBUGS-3949)

Regional-DR is not supported in environments deployed on IBM Z

Regional-DR is not supported in OpenShift Data Foundation environments deployed on IBM Z because ACM 2.15 is not supported on this platform for this release. This impacts both new and upgraded deployments on IBM Z.

(This content is not included.DFBUGS-5369)

Node crash results in kubelet service failure causing Data Foundation in error state

An unexpected node crash in an OpenShift cluster might lead to node being stuck in NotReady state and affect storage cluster.

Workaround:

Get the pending CSR:

$ oc get csr | grep Pending

Approve the pending CSR:

$ oc adm certificate approve <csr>

(This content is not included.DFBUGS-3636)

CIDR range does not persist in csiaddonsnode object when the respective node is down

When a node is down, the Classless Inter-Domain Routing (CIDR) information disappears from the csiaddonsnode object. This impacts the fencing mechanism when it is required to fence the impacted nodes.

Workaround: Collect the CIDR information immediately after the NetworkFenceClass object is created.

(This content is not included.DFBUGS-2948)

After node replacement, new mon pod is failing to schedule

After node replacement, the new mon pod fails to schedule itself in the newly added node. As a result, mon pod is stuck in the Pending state, which impacts the storagecluster status with a mon being unavailable.

Workaround: Manually update the new mon deployment with the correct nodeSelector.

(This content is not included.DFBUGS-2918)

ceph df reports an invalid MAX AVAIL value when the cluster is in stretch mode

When a CRUSH rule in a Red Hat Ceph Storage cluster has multiple take steps, the ceph df report shows the wrong maximum available size for associated pools.

(This content is not included.DFBUGS-1748)

DRPCs protect all persistent volume claims created on the same namespace

The namespaces that host multiple disaster recovery (DR) protected workloads protect all the persistent volume claims (PVCs) within the namespace for each DRPlacementControl resource in the same namespace on the hub cluster that does not specify and isolate PVCs based on the workload using its spec.pvcSelector field.

This results in PVCs that match the DRPlacementControl spec.pvcSelector across multiple workloads. Or, if the selector is missing across all workloads, replication management to potentially manage each PVC multiple times and cause data corruption or invalid operations based on individual DRPlacementControl actions.

Workaround: Label PVCs that belong to a workload uniquely, and use the selected label as the DRPlacementControl spec.pvcSelector to disambiguate which DRPlacementControl protects and manages which subset of PVCs within a namespace. It is not possible to specify the spec.pvcSelector field for the DRPlacementControl using the user interface, hence the DRPlacementControl for such applications must be deleted and created using the command line.

Result: PVCs are no longer managed by multiple DRPlacementControl resources and do not cause any operation and data inconsistencies.

(This content is not included.DFBUGS-1749)

Disabled PeerReady flag prevents changing the action to Failover

The DR controller executes full reconciliation as and when needed. When a cluster becomes inaccessible, the DR controller performs a sanity check. If the workload is already relocated, this sanity check causes the PeerReady flag associated with the workload to be disabled, and the sanity check does not complete due to the cluster being offline. As a result, the disabled PeerReady flag prevents you from changing the action to Failover.

Workaround: Use the command-line interface to change the DR action to Failover despite the disabled PeerReady flag.

(This content is not included.DFBUGS-665)

Ceph becomes inaccessible and IO is paused when connection is lost between the two data centers in stretch cluster

When two data centers lose connection with each other but are still connected to the Arbiter node, there is a flaw in the election logic that causes an infinite election among Ceph Monitors. As a result, the Monitors are unable to elect a leader and the Ceph cluster becomes unavailable. Also, IO is paused during the connection loss.

Workaround: Shutdown the monitors of any one data zone by bringing down the zone nodes. Additionally, you can reset the connection scores of surviving Monitor pods.

As a result, Monitors can form a quorum and Ceph becomes available again and IOs resumes.

(This content is not included.DFBUGS-425)

RBD applications fail to Relocate when using stale Ceph pool IDs from replacement cluster

For the applications created before the new peer cluster is created, it is not possible to mount the RBD PVC because when a peer cluster is replaced, it is not possible to update the CephBlockPoolID’s mapping in the CSI configmap.

Workaround: Update the rook-ceph-csi-mapping-config configmap with cephBlockPoolID’s mapping on the peer cluster that is not replaced. This enables mounting the RBD PVC for the application.

(This content is not included.DFBUGS-527)

Information about lastGroupSyncTime is lost after hub recovery for the workloads which are primary on the unavailable managed cluster

Applications that are previously failed over to a managed cluster do not report a lastGroupSyncTime, thereby causing the trigger of the alert VolumeSynchronizationDelay. This is because when the ACM hub and a managed cluster that are part of the DRPolicy are unavailable, a new ACM hub cluster is reconstructed from the backup.

Workaround: If the managed cluster to which the workload was failed over is unavailable, you can still failover to a surviving managed cluster.

(This content is not included.DFBUGS-376)

MCO operator reconciles the veleroNamespaceSecretKeyRef and CACertificates fields

When the OpenShift Data Foundation operator is upgraded, the CACertificates and veleroNamespaceSecretKeyRef fields under s3StoreProfiles in the Ramen config are lost.

Workaround: If the Ramen config has the custom values for the CACertificates and veleroNamespaceSecretKeyRef fields, then set those custom values after the upgrade is performed.

(This content is not included.DFBUGS-440)

For discovered apps with CephFS, sync stop after failover

For CephFS-based workloads, synchronization of discovered applications may stop at some point after a failover or relocation. This can occur with a Permission Denied error reported in the ReplicationSource status.

Workaround:

Remove DR option is not available for discovered apps on the Virtual machines page

The Remove DR option is not available for discovered applications listed on the Virtual machines page.

Workaround:

DR Status is not displayed for discovered apps on the Virtual machines page

DR Status is not displayed for discovered applications listed on the Virtual machines page.

Workaround:

PVCs deselected after failover doesn’t cleanup the stale entries in the secondary VRG causing the subsequent relocate to fail

If PVCs were deselected after a workload failover, and a subsequent relocate operation is performed back to the preferredCluster, stale PVCs may still be reported in VRG. As a result, the DRPC may report its Protected condition as False, with a message similar to the following:

VolumeReplicationGroup (/) on cluster is not reporting any lastGroupSyncTime as primary, retrying till status is met.

Workaround:

To resolve this issue, manually clean up the stale PVCs (that is, those deselected after failover) from VRG status.

Secondary PVCs aren’t removed when DR protection is removed for discovered apps

On the secondary cluster, CephFS PVCs linked to a workload are usually managed by the VolumeReplicationGroup (VRG). However, when a workload is discovered using the Discovered Applications feature, the associated CephFS PVCs are not marked as VRG-owned. As a result, when the workload is disabled, these PVCs are not automatically cleaned up and become orphaned.

Workaround: To clean up the orphaned CephFS PVCs after disabling DR protection for a discovered workload, manually delete them using the following command:

$ oc delete pvc <pvc-name> -n <pvc-namespace>

(This content is not included.DFBUGS-2827)

DRPC in Relocating state after minor upgrade

After upgrading from version 4.19 to 4.20, the DRPC (Disaster Recovery Placement Control) may enter a Relocating state. During this process, a new VGR (VolumeGroupReplication) is created with a different naming convention, resulting in two VGRs attempting to claim the same PVC. This conflict can cause temporary instability in the DRPC status.

Workaround: Delete the old VGR (the one with the previous naming convention). The new VGR will then successfully claim the PVC, and the DRPC will return to a healthy state after some time.

(This content is not included.DFBUGS-4450)

Ceph in warning state after adding capacity to cluster

After device replacement or capacity addition it is observed that Ceph is in HEALTH_WARN state with mon reporting slow ops. However, there is no impact to the usability of the cluster.

(This content is not included.DFBUGS-1273)

OSD pods restart during add capacity

OSD pods restart after performing cluster expansion by adding capacity to the cluster. However, no impact to the cluster is observed apart from pod restarting.

(This content is not included.DFBUGS-1426)

Sync stops after PVC deselection

When a PersistentVolumeClaim (PVC) is added to or removed from a group by modifying its label to match or unmatch the group criteria, sync operations may unexpectedly stop. This occurs due to stale protected PVC entries remaining in the VolumeReplicationGroup (VRG) status.

Workaround:

Manually edit the VRG’s status field to remove the stale protected PVC:

$ oc edit vrg <vrg-name> -n <vrg-namespace> --subresource=status

(This content is not included.DFBUGS-4012)

PVs Remain Stuck in Released State After Workload Deletion

After the final sync, all temporary PVs/PVCs are deleted; however, for some PVs, the persistentVolumeReclaimPolicy remains set to Retain, causing the PVs to stay in the Released state.

Workaround:

Edit the PVs persistentVolumeReclaimPolicy using the command:

$ oc edit pv <pv-name>

Change persistentVolumeReclaimPolicy to Delete. The stuck PVs will disappear.

(This content is not included.DFBUGS-4535)

Unable to create new OBCs using Noobaa

When provisioning an NSFS bucket via ObjectBucketClaim (OBC), the default filesystem path is expected to use the bucket name. However, if path is set in OBC.Spec.AdditionalConfig, it should take precedence. This behavior is currently inconsistent, resulting in failures when creating new OBCs.

(This content is not included.DFBUGS-3817)

Poor CephFS performance on stretch clusters

Workloads with many small metadata operations might exhibit poor performance because of the arbitrary placement of metadata server pods (MDS) on multi-site Data Foundation clusters.

(This content is not included.DFBUGS-1753)

SELinux relabelling issue with a very high number of files

When attaching volumes to pods in Red Hat OpenShift Container Platform, the pods sometimes do not start or take an excessive amount of time to start. This behavior is generic and it is tied to how SELinux relabelling is handled by Kubelet. This issue is observed with any filesystem based volumes having very high file counts. In OpenShift Data Foundation, the issue is seen when using CephFS based volumes with a very high number of files. There are multiple ways to work around this issue. Depending on your business needs you can choose one of the workarounds from the knowledgebase solution https://access.redhat.com/solutions/6221251.

(This content is not included.Jira#3327)

Orphaned CSIAddonsNode CRs leading to errant sidecar connection attempts by the csi-addons-controller-manager pod

When a worker is deleted, a stale CSIAddonsNode owned by the DaemonSet remains in the cluster. This results in false connection attempts by csi-addons to an endpoint that no longer exists.

Workaround: Manually identify and delete the CSIAddonsNode resources linked to the removed worker node.

# oc get csiaddonsnodes --no-headers | awk '{print $1}' | grep -v -f <(oc get nodes --no-headers | awk '{print $1}') | xargs -r oc delete csiaddonsnode

(This content is not included.DFBUGS-4466)

UI shows "Unauthorized" error and Blank screen with loading temporarily during ODF operator installation

During OpenShift Data Foundation operator installation, sometimes the InstallPlan transiently goes missing which causes the page to show unknown status. This does not happen regularly. As a result, the messages and title go missing for a few seconds.

(This content is not included.DFBUGS-3574)

Optimize DRPC creation when multiple workloads are deployed in a single namespace

When multiple applications refer to the same placement, then enabling DR for any of the applications enables it for all the applications that refer to the placement.

If the applications are created after the creation of the DRPC, the PVC label selector in the DRPC might not match the labels of the newer applications.

Workaround: In such cases, disabling DR and enabling it again with the right label selector is recommended.

(This content is not included.DFBUGS-120)

ODF-CLI tools misidentify stale volumes

Stale subvolume CLI tool misidentifies the valid CephFS persistent volume claim (PVC) as stale due to an issue in the stale subvolume identification tool. As a result, stale subvolume identification functionality will not be available till the issue is fixed.

(This content is not included.DFBUGS-3778)