Removing Orphaned Cephfs Subvolumes and Snapshots in ODF

Solution Verified - Updated

Environment

Red Hat OpenShift Container Platform (OCP) 4.14+
Red Hat OpenShift Data Foundation (ODF) 4.14+
Red Hat Ceph Storage (RHCS) 6+

Issue

Sometimes Cephfs subvolumes and snapshots become orphaned in ODF. That is to say, Cephfs subvolumes and snapshots lose their connection to their requisite PV in Openshift. The result is that ODF ends up housing more data than it should. This data cannot be accessed by Openshift, making it essentially useless. The following document is an effort to describe how to check for and remove cephfs snapshots and subvolumes.

Resolution

This solution focuses only on cleaning Orphaned Cephfs Subvolumes and Snapshots.

To implement strategies to prevent the issue from happening in the first place: CephFS VolumeSnapshot creation (and restore) takes a very long time to complete.

NOTE: Use this process at your own risk. Removal of cephfs subvolumes or snapshots that are not orphans can result in data loss. If at any point you are confused by this process or run into any errors you do not understand, please contact Red Hat Support.

Checking For and Removing Cephfs Snapshots

  1. Get a list of your Cephfs snapshots and their requisite csi-volume names in Openshift, and save it to a file.
# oc get volumesnapshotcontent -o 'custom-columns=Name:.spec.volumeSnapshotRef.name,PARENT-CSI-VOL+CEPH-CSI-VOL-SUFFIX-ID:.spec.source.volumeHandle,SNAP-SUFFIX-ID:.status.snapshotHandle,SNAP-CLASS-NAME:.spec.volumeSnapshotClassName' | sed -n '1p; /cephfs/p' > /tmp/cephfs_volumesnapshots.txt
  1. Get a list of all cephfs snapshots in Ceph. First rsh into the rook-ceph-tools pod and run the following command:
# for i in `ceph fs subvolume ls ocs-storagecluster-cephfilesystem csi | jq -r '.[].name'` ; do echo subvolume $i: ; echo -----; ceph fs subvolume snapshot ls ocs-storagecluster-cephfilesystem $i csi; echo ----- ; done > /tmp/snaplist.txt
  1. Exit the pod, then copy the list of snapshots from the rook-ceph-tools pod to your local /tmp directory:
# exit
# oc exec -n openshift-storage $(oc get pods -l app=rook-ceph-tools --no-headers -n openshift-storage | awk '{print $1}') -- cat /tmp/snaplist.txt > /tmp/snaplist.txt
  1. Compare both files and output only the orphans and it's parent to a new file called orphaned_snapshots.txt:
awk 'NR>1 {print substr($3, length($3)-35)}' /tmp/cephfs_volumesnapshots.txt | sort | grep -v -f - /tmp/snaplist.txt | grep -o 'csi-snap-[a-f0-9-]\{36\}' | awk 'NR>1 {print substr($3, length($3)-35)}' /tmp/cephfs_volumesnapshots.txt | sort | grep -v -f - /tmp/snaplist.txt | grep -o 'csi-snap-[a-f0-9-]\{36\}' | awk '
BEGIN { while ((getline pattern < "/dev/stdin") > 0) patterns[pattern] = 1 }
/^subvolume/ { subvol = $0 }
{ for (pattern in patterns) if ($0 ~ pattern) { print subvol; print $0; echo; break } }
' /tmp/snaplist.txt > /tmp/orphaned_snapshots.txt
  1. All orphaned snapshots are listed in orphaned_snapshots.txt, the orphaned snapshot's parent is listed directly above it (if nothing is listed in this file, you have no orphans):
subvolume csi-vol-26ff197a-25e6-4b67-bd3f-0a6fc55299fe:         <-------- Parent of csi-snap-f31ccf25-5976-42c0-8b18-dc897f9be650
        "name": "csi-snap-f31ccf25-5976-42c0-8b18-dc897f9be650" <-------- Orphaned Snapshot
subvolume csi-vol-788285d0-a4ab-47a6-bc25-c16ae4649168:         <-------- Parent of csi-snap-393e63a0-aa78-421b-a378-b34f4b22290b  
        "name": "csi-snap-393e63a0-aa78-421b-a378-b34f4b22290b" <-------- Orphaned Snapshot
  1. To get rid of the orphaned snapshots, you need to issue the following command in the rook-ceph-tools pod for each of the listed orphans:
# ceph fs subvolume snapshot rm ocs-storagecluster-cephfilesystem ${subvolume} ${snap} --group_name csi

Replace ${subvolume} with the parent cephfs subvolume of the orphaned snapshot, and {snap} with the snapshot name. For example, to remove csi-snap-f31ccf25-5976-42c0-8b18-dc897f9be650:

# ceph fs subvolume snapshot rm ocs-storagecluster-cephfilesystem csi-vol-26ff197a-25e6-4b67-bd3f-0a6fc55299fe csi-snap-f31ccf25-5976-42c0-8b18-dc897f9be650 --group_name csi
  1. Enact Step 6 for each of the orphaned snapshots listed in orphaned_snapshots.txt

Checking For and Removing Cephfs Subvolumes

  1. Get a list of all cephfs PVs in Openshift and their requisite volumeHandle attributes, and save it to a file.:
# oc get pv -o 'custom-columns=NAME:.spec.claimRef.name,PVNAME:.metadata.name,STORAGECLASS:.spec.storageClassName,VOL-NAME:.spec.csi.volumeHandle'  | grep cephfs > /tmp/cephfs_pvs.txt
  1. Get a list of all cephfs subvolumes in Ceph. First rsh into the rook-ceph-tools pod and run the following command:
# ceph fs subvolume ls ocs-storagecluster-cephfilesystem csi | jq -r '.[].name' > /tmp/cephfs_subvolumes.txt
  1. Exit the pod, then copy the list of cephfs subvolumes from the rook-ceph-tools pod to your local /tmp directory:
# oc exec -n openshift-storage $(oc get pods -l app=rook-ceph-tools --no-headers -n openshift-storage | awk '{print $1}') -- cat /tmp/cephfs_subvolumes.txt > /tmp/cephfs_subvolumes.txt
  1. Compare both files and output only the orphaned subvolumes to a new file called orphaned_subvolumes.txt:
grep -v "$(awk '{print $4}' /tmp/cephfs_pvs.txt | grep -o '[0-9a-f]\{8\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{4\}-[0-9a-f]\{12\}')" /tmp/cephfs_subvolumes.txt > /tmp/orphaned_subvolumes.txt
  1. All orphaned cephfs subvolumes are listed in orphaned_subvolumes.txt (if nothing is listed in this file, you have no orphans):
# cat /tmp/orphaned_subvolumes.txt 
csi-vol-806f4943-cf3b-48c2-b29d-9f544e01fbaa <-------- Orphaned Cephfs Subvolume
csi-vol-1489c09e-e11b-4160-9836-e6fc36e43075 <-------- Orphaned Cephfs Subvolume
  1. To get rid of the orphaned subvolumes, you need the to issue the following command in the rook-ceph-tools pod for each of the listed orphans:
# ceph fs subvolume subvolumes rm ocs-storagecluster-cephfilesystem ${subvolume} --group_name csi

Replace ${subvolume} with a subvolume name from orphaned_subvolumes.txt. For example, to remove csi-vol-806f4943-cf3b-48c2-b29d-9f544e01fbaa:

# ceph fs subvolume rm ocs-storagecluster-cephfilesystem csi-vol-806f4943-cf3b-48c2-b29d-9f544e01fbaa --group_name csi
  1. Enact Step 6 for each of the orphaned subvolumes listed in orphaned_subvolumes.txt

Root Cause

There are many reasons that orphaned cephfs subvolumes and cephfs snapshots can become orphaned. The most common is backup solutions (OADP, Velero, Kasten, etc.) have timeout values that are set too low. If you are encountering this issue frequently and use any of the stated backup solutions, please reference [1]. It goes over why this happens, and how to correct it.

[1] https://access.redhat.com/solutions/7128329

SBR

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.