How to share the same CephFS volume across two external ODF clusters ?

Note: Ensure before proceeding that the same CephFS subvolumegroup is configured in both the OCP clusters from the same external Ceph cluster.

1. Create Cephfs Storage Class (optional), if it doesnt exists:
   Note:
   A dedicated storage class with the reclaimPolicy as Retain and VolumeBindingMode as WaitForFirstConsumer is recommended.
   This is not mandatory, but it is a way to prevent the volume deletion by accident or the accidental bound between the PVC with an existent or automatic created PV.

   cluster-01 # echo "
   kind: StorageClass
   apiVersion: storage.k8s.io/v1
   metadata:
     name: manual-shared-cephfs-volume
   provisioner: openshift-storage.cephfs.csi.ceph.com
   parameters:
     csi.storage.k8s.io/provisioner-secret-namespace: openshift-storage
     csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
     csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
     mounter: fuse
     fsName: ocs-storagecluster-cephfilesystem
     csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
     clusterID: openshift-storage
     csi.storage.k8s.io/controller-expand-secret-namespace: openshift-storage
     csi.storage.k8s.io/node-stage-secret-namespace: openshift-storage
   reclaimPolicy: Retain
   allowVolumeExpansion: true
   volumeBindingMode: WaitForFirstConsumer " |  oc create -f  -
   

2. Create the namespace and deployment
   2.1. Two external ODF clusters will be used in this example, we’ll use ns-01 namespace on both clusters:

       cluster-01 # oc create ns ns-01; oc project ns-01
   
       cluster-02 # oc create ns ns-01; oc project ns-01
   

   2.2. Collect the sa.scc.mcs and scc.supplemental-groups from cluster-01. These values will later be used for the SCC and securityContext on cluster-02:

   cluster-01 # oc describe project ns-01 | egrep 'sa.scc.mcs|scc.supplemental-groups'
        openshift.io/sa.scc.mcs=s0:c27,c24
        openshift.io/sa.scc.supplemental-groups=1000750000/10000
   

   2.3. Create a new application:

   cluster-01 # oc new-app --name  deploy-01  registry.access.redhat.com/rhscl/httpd-24-rhel7
   

3. Create PVC/PV
   3.1. This first PVC could be requested in any available way since it is created dynamically:

   cluster-01 # oc set volumes deployment/deploy-01 --claim-name base-pvc-for-share --claim-size=1G --claim-class=ocs-external-storagecluster-cephfs --claim-mode=rwm --mount-path /test1 --add
   

   3.2. Check if the PVC is bounded to the PV:

       cluster-01 # oc get pvc 	 -w
   
       cluster-01 # PV_NAME=$(oc get pvc base-pvc-for-share -o jsonpath='{.spec.volumeName}')
   

4. Extract PV information for another ODF cluster:
   4.1. Extract the PV. Using grep -v will help to remove some unnecessary fields from the yaml:

   cluster-01 # oc get pv $PV_NAME -o yaml | egrep -v " uid:| resourceVersion:|  creationTimestamp:| kubernetes.io/pv-protection| pv.kubernetes.io" > PV-test-02.yaml
   

   4.1.1. The following fields must be updated:

   apiVersion: v1
   kind: PersistentVolume
   metadata:
     name: cephfs-test-02                                           ## New name for PV
   spec:
     claimRef:
       name: cephfs-test-02                                         ## New name for PVC
       namespace: ns-01                                             ## Change this if required
     csi:
     .....
       storageClassName: ocs-external-storagecluster-cephfs
       persistentVolumeReclaimPolicy: Retain                        ## Changed from Delete
   

   SECONDARY CLUSTER:
   4.2. Create the PV on the secondary cluster. The status should be Available until the creation of the PVC:

       cluster-02 # oc create -f  PV-test-02.yaml
   
       cluster-02 # oc get pv cephfs-test-02 -w       # Ensure the status says Available
   

   4.3. Create the deployment of application and attach a PVC that will be using the shared PV from primary ODF cluster:

       cluster-02 # oc new-app --name  deploy-02  registry.access.redhat.com/rhscl/httpd-24-rhel7
   
       cluster-02 # oc set volumes deployment/deploy-02 --claim-name cephfs-test-02 --claim-size=1G --claim-class=ocs-external-storagecluster-cephfs --claim-mode=rwm --mount-path /test2 --add
   

   4.4. Security Context
   4.4.1. The shared configuration is done, but the permissions must be adjusted with values from step 2.2.
   In order to do so, a custom SCC and serviceaccount must be created. The deploy (of any kind: deployment, deploymentconfig, cronjob, etc.) must be set to use the serviceaccount created.
   The SCC allows the usage of any seLinuxContext and supplementalGroups (sa.scc.supplemental-groups) from the namespace of the first PVC, in this example test-01.
   This SCC also restricts the execution to no-root users. If the usage of root is necessary, this must be adjusted:

   cluster-02 # echo "
   apiVersion: security.openshift.io/v1
   fsGroup:
     type: RunAsAny
   kind: SecurityContextConstraints
   metadata:
     name: cephfs-share-scc
   priority: null
   readOnlyRootFilesystem: false
   AllowPrivileged: false
   requiredDropCapabilities: null
   seLinuxContext:
     type: RunAsAny
   runAsUser:
     type: MustRunAsNonRoot
   supplementalGroups:
     type: MustRunAs
     ranges:
     - min: 1000750000             ## Get it from step 2.2
       max: 1000750001             ## Just add 1 to above value
   seccompProfiles:
   - '*'
   users:
   - system:serviceaccount:ns-01:cephfs-share-sa
   volumes:
   - '*' " | oc create -f -
   

   4.4.2. Create the serviceaccount:

       cluster-02 # oc create serviceaccount cephfs-share-sa
   
       cluster-02 # oc adm policy add-scc-to-user cephfs-share-scc system:serviceaccount:ns-01:cephfs-share-sa
   

   4.4.3. Set the deploy's securityContext on cluster-02:

   kind: Deployment
   apiVersion: apps/v1
   spec:
   ....
     template:
     ...
      spec:
      ...
       securityContext:
         fsGroup: 1000750000                  ## Get it from step 2.2
         seLinuxOptions:
           level: 's0:c27,c24'	               ## Get it from step 2.2
   		seviceAccountName: cephfs-share-sa
   

   Add PVC to the above deployment and create it

   4.5. Since the PV already has the related PVC and namespace defined, the status of the PV and PVC should change to Bound:

   cluster-02 # oc get pv,pvc cephfs-test-02
   

   Note: For deletion of the shared PV:
   On cluster-01, the PV's YAML shows persistentVolumeReclaimPolicy is set to Delete.
   However, when this manifest was copied and applied to cluster-02, the reclaim policy was changed to Retain.
   This change prevents the underlying subvolume from being cleaned up if the PV is deleted on any cluster other than the primary cluster (cluster-01 in this case).
   While this approach is effective, it is crucial to note that the PV on the primary cluster must be deleted last. Only then will the subvolume within Ceph be cleaned up.

Related Articles:
How to share the same volume between namespaces using ODF 4