Finding and removing orphaned RBD images in ODF and OCS 4.x

Solution Verified - Updated

Environment

  • Red Hat OpenShift Date Foundation 4.9 and above

Issue

  • Orphaned RBD images occupy the space even after the PVC and PV has been deleted
  • When checking the number of existing images within Ceph on ODF, sometimes we see a mismatch between the number of RBD images versus the number of existing PVs/PVCs on OCP, where the number of RBD images is higher than the number of created and active PVs/PVCs, evidencing these images are orphaned somehow. So how might one delete these orphaned images not linked to any PV/PVC?

Resolution

Important

  • Please investigate how did cluster reached the current state before attempting to delete any orphaned volumes. The volume delete flow in ODF should prevent any orphaned volumes. A Bugzilla should be filed for further investigation if required.
  • Please perform the orphan removal with extreme caution; wrongly deleting any RBD image that is in use will result in unrecoverable data loss
  • Please open an RH support case to get technical guidance before you perform this on a production cluster

The resolution involves finding the RBD volumes and removing them from the cluster manually.

Pre-requisite: Configure the ceph toolbox pod as documented here; this will help us to run all the relevant ceph commands

  1. Check the available number of images in RBD for the specific pool, keep a note of the number of images
  • For this article, the default ocs-storagecluster-cephblockpool pool is used. List the images in this pool using #rbd ls command. You may redirect the output to a file.
$ oc exec -n openshift-storage $(oc get pod -n openshift-storage -l app=rook-ceph-tools -o name) -- rbd -p ocs-storagecluster-cephblockpool ls |tee /tmp/all_rbd_images.txt
  1. List the number of volumes/PV in OpenShift; you may redirect it to a file. If you have more than one storageclass for RBD volumes then please check available volumes for all RBD storage classes.
$ oc get pv -o 'custom-columns=IMAGENAME:.spec.csi.volumeAttributes.imageName'  | grep -v none | grep -v IMAGENAME |tee /tmp/all_pvs.txt
  1. Compare both files using comm. This output will tell you rbd images that are uniq to all_rbd_images.txt that do not exist in sort all_pvs.txt
$ comm -23 <(/usr/bin/sort all_rbd_images.txt) <(/usr/bin/sort all_pvs.txt) > /tmp/rbd_images_with_no_PV.txt
  1. Some of these rbd images can be snapshots that can be associated to a VolumeSnapshotContent (VSC). We must not delete an rbd image associated to a VolumeSnapshotContent.
    So we need to exclude these from above list rbd_images_with_no_PV.txt --> create a subset called: rbd_images_with_no_PV_and_no_VSC.txt

Example to manually identify rbd images associated to a VSC

csi-snap-0ee7cc43-8ec2-42bd-af10-807b3dcddc21 has no PV associated, but is related to this VSC:

snapcontent-3af20138-a416-4bd4-a2d9-c913f7e59918   true         32212254720   Delete           openshift-storage.rbd.csi.ceph.com   ocs-storagecluster-rbdplugin-snapclass   rhel8-cd97caf10611             openshift-virtualization-os-images   15d

% oc get volumesnapshotcontent -o yaml|grep 0ee7cc43-8ec2-42bd-af10-807b3dcddc21                         
    snapshotHandle: 0001-0011-openshift-storage-000000000000000b-0ee7cc43-8ec2-42bd-af10-807b3dcddc21

% oc get volumesnapshotcontent -o yaml|grep 0ee7cc43-8ec2-42bd-af10-807b3dcddc21 -B30|grep "name: snapcontent"
    name: snapcontent-a6f2a49c-0926-45dd-80f9-3f00afd6f494

Creation of the new list:

% cut -d "-" -f3,4,5,6,7 rbd_images_with_no_PV.txt  > ID_rbd_images_with_no_PV.txt

% for ID in $(cat ID_rbd_images_with_no_PV.txt); do
    echo "Checking image: $ID"
    snap_output=$(oc get volumesnapshotcontent -o yaml|grep $ID)
    if [ -z "$snap_output" ]; then
        echo "$ID" >> ID_rbd_images_with_no_PV_and_no_VSC.txt
    fi
done

% for ID in $(cat ID_rbd_images_with_no_PV_and_no_VSC.txt); do grep $ID rbd_images_with_no_PV.txt ; done > rbd_images_with_no_PV_and_no_VSC.txt
  1. From above list we need to find if these are parent of a child:
  • An rbd with a child must not be deleted.
  • A child is a clone from a snapshot.
  • Data of that child is dependent on the parent, which means parent cannot be deleted.
    NOTE. To make the child independent, we would need to flatten the clone (the child), but this is not required here.
    The "rbd children" command will display children of an image or its snapshot, see Content from docs.ceph.com is not included.Listing Children of a Snapshot
    If it reports nothing, that means it not a parent.
sh-5.1$ rbd -p ocs-storagecluster-cephblockpool children csi-vol-062ec147-ced7-4cf0-b259-38787b6ea347  
ocs-storagecluster-cephblockpool/csi-vol-61a7319b-09f2-45ba-95ca-62e1c7258910-temp  <<<-- children

To identify orphaned images, we use 'rbd children' command on each of the rbd_images_with_no_PV_and_no_VSC.txt as an additional check. If no children exist, we deem it orphaned:

% for image in $(cat rbd_images_with_no_PV_and_no_VSC.txt); do
    echo "Checking image: $image"
    children_output=$(oc exec -n openshift-storage $(oc get pod -n openshift-storage -l app=rook-ceph-tools -o name) -- rbd -p ocs-storagecluster-cephblockpool children $image)
    if [ -z "$children_output" ]; then
        echo "$image" >> rbd_images_with_no_PV_no_VSC_and_no_children.txt
    fi
done

  1. You can also map the persistent volumes in RBD storageclass to Ceph RBD images by following the steps here

  2. Once you have the list of RBD orphan volumes (rbd_images_with_no_PV_no_VSC_and_no_children.txt) , confirm that the volume is not in use from the Ceph side. There should be no watchers for the volume (ie: 'Watchers: none')

$ for i in $(cat /tmp/rbd_images_with_no_PV_no_VSC_and_no_children.txt);do echo $i; rbd -p ocs-storagecluster-cephblockpool status $i;done
    Watchers:
    	watcher=10.129.2.1:0/1849604752 client.4740 cookie=18446462598732840961   <--- There are active watchers for this volume and may be in use by an application.

NOTE. You can see how much data each rbd image contains with command below:

sh-5.1$ rbd -p ocs-storagecluster-cephblockpool du   
NAME                                                                                         PROVISIONED  USED    
csi-snap-0ee7cc43-8ec2-42bd-af10-807b3dcddc21@csi-snap-0ee7cc43-8ec2-42bd-af10-807b3dcddc21       30 GiB       0 B
csi-snap-0ee7cc43-8ec2-42bd-af10-807b3dcddc21                                                     30 GiB       0 B
csi-vol-062ec147-ced7-4cf0-b259-38787b6ea347                                                      30 GiB       0 B
csi-vol-062ec147-ced7-4cf0-b259-38787b6ea347-temp                                                 30 GiB   3.9 GiB
csi-vol-126aa626-b1ee-4d53-8793-2e7d75f755cb                                                     120 GiB       0 B
csi-vol-17875e43-9f3d-4660-aee2-28238885a247                                                      30 GiB   1.1 GiB

  1. If there are active watchers then you will need to trace the IP and client application. Oftentimes, these active watcher entries are stale, but it will prevent RBD volumes from being deleted with error "rbd: error: image still has watchers". Please refer this KCS article Ceph - rbd image cannot be deleted with "rbd: error: image still has watchers" to know how to unmap and clear the watcher list to proceed with deletion

  2. In ODF environment, you have to run the 'showmapped' command from the csi-rbdplugin pod where the PV is attached to:

Example:
$ oc rsh -n openshift-storage -c csi-rbdplugin pod/csi-rbdplugin-<pod-name>
$ rbd showmapped
id  pool                              namespace  image                                         snap  device   
0   ocs-storagecluster-cephblockpool             csi-vol-09477143-85e0-4c2f-bed1-f236d39aa9f3  -     /dev/rbd0
  • If you are not able to figure out on which worker node the image is mapped and mounted, then you may have to do this from all the rbdplugin pods
    You can use a bash script like :
$ for rbdpluginpods in $(oc get pod -l app=csi-rbdplugin  -o name); do echo $rbdpluginpods; oc exec -it $rbdpluginpods -c csi-rbdplugin rbd showmapped; done
  • This will list all the mapped RBD images from all the worker nodes. Identify in which csi-rbdplugin the orphaned RBD image is mapped, then get inside the corresponding csi-rbdplugin pod and run 
$ oc rsh -n openshift-storage -c csi-rbdplugin pod/csi-rbdplugin-<pod-name>
$ rbd unmap <imagename> -p ocs-storagecluster-cephblockpool
  1. Remove the orphaned RBD volumes
$ rbd -p ocs-storagecluster-cephblockpool rm csi-vol-e097a3e7-1098-11ec-ae7f-0a580a83002e
$ rbd -p ocs-storagecluster-cephblockpool rm csi-vol-062ec147-ced7-4cf0-b259-38787b6ea347-temp

NOTE. We may fail to delete a ceph-rbd image (that comes from a VSC) with this error:

sh-5.1$ rbd -p ocs-storagecluster-cephblockpool rm csi-snap-9d50d946-c910-42c5-8481-cac963aebcc7
Removing image: 0% complete...failed.
rbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed.
sh-5.1$

In this case, we need to purge the snapshot, and then delete the image:

rbd -p ocs-storagecluster-cephblockpool snap purge csi-snap-9d50d946-c910-42c5-8481-cac963aebcc7
rbd -p ocs-storagecluster-cephblockpool rm csi-snap-9d50d946-c910-42c5-8481-cac963aebcc7

Root Cause

  • The volume delete flow in ODF should prevent any orphaned volumes. But in some corner cases, we have seen the presence of orphaned RBD volumes in the cluster. Some scenarios include:
  • Manual deletion of PV before the PVC.
  • PVC reclaimpolicy is set to retain instead of delete. This will prevent the underlying RBD image from being deleted.
  • When the volumesnapshotclass "ocs-storagecluster-rbdplugin-snapclass" has configured the DELETIONPOLICY to Retain, the volumesnapshotcontents (VSC) will have the deletionPolicy set to Retain. In that case, when the VSC is deleted the underlying ceph-rbd image "csi-snap" is not deleted, neither the associated snapshot of the csi-snap image.

Diagnostic Steps

  • Follow the first few steps in the Resolution field to diagnose if any orphaned RBD images exist in the ODF cluster.

  • Another simpler approach would be to run the following command directly inside the rook-ceph-toolbox pod:

$ rbd ls -p ocs-storagecluster-cephblockpool --format json | jq -r '.[] | select(.referenced == false) | .name'
  • Example of a csi-snap images of a VSC and its associated ceph-rbd snapshot with the same name as csi-snap image
sh-5.1$ rbd -p ocs-storagecluster-cephblockpool info csi-snap-9d50d946-c910-42c5-8481-cac963aebcc7
rbd image 'csi-snap-9d50d946-c910-42c5-8481-cac963aebcc7':
        size 160 GiB in 40960 objects
        order 22 (4 MiB objects)
        snapshot_count: 1
        id: 7f69d9ad5d9f5f
        block_name_prefix: rbd_data.7f69d9ad5d9f5f
        format: 2
        features: layering, deep-flatten, operations
        op_features: clone-child
        flags:
        create_timestamp: Wed Oct  8 08:34:27 2025
        access_timestamp: Wed Oct  8 08:34:27 2025
        modify_timestamp: Wed Oct  8 08:34:27 2025
        parent: ocs-storagecluster-cephblockpool/csi-vol-96020363-b5ab-467b-b6cb-a4401c020d0d@dfa92c7b-fe07-4772-ab78-b52c7c82b46c
        overlap: 160 GiB

The ceph-rbd image "csi-snap-9d50d946-c910-42c5-8481-cac963aebcc7" has one snapshot called "csi-snap-9d50d946-c910-42c5-8481-cac963aebcc7"

sh-5.1$ rbd -p ocs-storagecluster-cephblockpool snap ls csi-snap-9d50d946-c910-42c5-8481-cac963aebcc7
SNAPID  NAME                                           SIZE     PROTECTED  TIMESTAMP
  4823  csi-snap-9d50d946-c910-42c5-8481-cac963aebcc7  160 GiB             Wed Oct  8 08:34:28 2025
sh-5.1$
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.