Validate Data Utilization for CephBlockPool/CephFS storageclass in Ceph and Correlate with PVC/Namespace

Environment

Red Hat OpenShift Container Platform (RHOCP) v4.x
Red Hat OpenShift Data Foundations (RHODF) v4.x
Red Hat OpenShift Container Storage (RHOCS) v4.x

Issue

In many instances, there may be a need to understand how much data in reference to a PV/PVC in OpenShift is being consumed in backend storage (Ceph). There are a few ways to accomplish this. This solution will cover the association of the image name (csi-vol) and its association with the PVC.

Resolution

Prerequisites:

Configuring the Rook-Ceph Toolbox in OpenShift Data Foundation 4.x

Command Line Interface (CLI):

RBD:

1. Run the following command and capture the csi-vol data associated with the PV.

$ oc get pv -o 'custom-columns=PVNAME:.metadata.name,PVCNAME:.spec.claimRef.name,NAMESPACE:.spec.claimRef.namespace,IMAGENAME:.spec.csi.volumeAttributes.imageName,STORAGECLASS:.spec.storageClassName'

2. rsh into the rook-ceph-tools pod and run one of the following commands to match the csi-vol information with data used:

Quick Look:

$ rbd -p ocs-storagecluster-cephblockpool du

More Verbose, takes longer to capture:

$ rbd -p ocs-storagecluster-cephblockpool du --exact

Note: rbd shows GiB, MiB, no need convert.

CephFS:

1. Run the following command and capture the csi-vol data associated with the PV.

$ oc get pv -o 'custom-columns=NAME:.spec.claimRef.name,NAMESPACE:.spec.claimRef.namespace,PVNAME:.metadata.name,STORAGECLASS:.spec.storageClassName,VOL-NAME:.spec.csi.volumeAttributes.subvolumeName'

2. rsh into the rook-ceph-tools pod and run one of the following commands to match the csi-vol information with data used:

List all CephFS csi-vols:

sh-5.1$ ceph fs subvolume ls ocs-storagecluster-cephfilesystem --group_name csi

[
    {
        "name": "csi-vol-<name>"
    }
]

Specify csi-vol by changing name to match desired csi-vol:

sh-5.1$ ceph fs subvolume info ocs-storagecluster-cephfilesystem --group_name csi csi-vol-<name> | grep bytes

Note: The output will show the following (convert from bytes to GiB or MiB):

"bytes_pcent": <----- Percentage Used
"bytes_quota": <----- Size of PVC in bytes
"bytes_used": <------ Usage of PVC in bytes

OCP Console GUI:

Regarding PVC usage, many PVC usage details can be shown in the OCP Console GUI under "Storage" -> "Persistent Volume Claims" -> if monitoring is enabled on that namespace:

$ oc label namespace <namespace> openshift.io/cluster-monitoring="true"

Another method is the Monitor Persistent Volume Claim (PVC) utilization in OpenShift solution. For PVC usage metrics, You can navigate to the Openshift Console GUI --> Observe --> Metrics --> Queries and enter the following queries, then hit "Run Query":

kubelet_volume_stats_used_bytes
kubelet_volume_stats_available_bytes 

The above will provide the PVC pod details on what exactly is being consumed per PVC.

Additional information can be found in the Monitoring OpenShift Data Foundation product documentation.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.