Migrate persistent data to another Storage Class in DG 8 Operator in OCP 4

Environment

• Red Hat Openshift Container Platform (OCP) 4
• Red Hat Data Grid Operator 8.x

Issue

The Persistent Volume Claim (PVC) migration of a Data Grid Infinispan cluster to a new Storage Class for a technical reason is required. Since a StatefulSet (SS) deploys the Infinispan cluster, changing the PVC names is impossible because a pattern configured in the SS sets that name. For that reason, the recreation of the PVC is needed.

In case of migration of PVs not attached on a StatefulSet follow the "How to migrate data between PVs in OpenShift 4"

Resolution

Backup all the data first before doing this procedure (via backup/restore).

The below procedure will create a temporary deployment pod, associate the pvc with it move the data, and then delete and recreate a new pvc with the same name/volume size - this is the migrate data, for stale data handling see the solution DG 8 recovering from Stale State inside pod, which include one extra step for deleting the PVC.

Also, it assumes the infinispan custom resource is called example-infinispan and it has two pods (example-infinispan-0 and example-infinispan-1), so its associated volumes will be respectively data-volume-example-infinispan-0 and data-volume-example-infinispan-1.

1. Move onto the Data Grid namespace:

   $ oc project datagrid-project
   

   ^ The project will be different (do oc get project to list all projects and oc project to verify name of current project).

2. Scale to zero the infinispan custom resource instance. In the example below, the Infinispan instance was called example-infinispan:

   $ oc patch infinispan example-infinispan --patch '{"spec":{"replicas":0}}' --type=merge
   

   ^ The infinispan custom resource will be different (do oc get infinispan to list all infinispan custom resource in the project).

3. Create a temporary deployment using the rhel-tools image and scale it to zero:

   $ oc create deployment sleep --image=registry.access.redhat.com/rhel7/rhel-tools -- tail -f /dev/null 
   $ oc scale deployment/sleep --replicas=0
   

4. Mount old PVCs of the Infinispan nodes to this new deployment. In the example, the Infinispan cluster called example-infinispan has two replicas:

   $ oc set volume deployment/sleep --add -t pvc --name=data-volume-example-infinispan-0 \
   --claim-name=data-volume-example-infinispan-0 \
   --mount-path=/data-volume-example-infinispan-0
   $ oc set volume deployment/sleep --add -t pvc --name=data-volume-example-infinispan-1 \
   --claim-name=data-volume-example-infinispan-1 \
   --mount-path=/data-volume-example-infinispan-1
   

   ^ Above is done for two pods/their pvc's (example-infinispan-0 and example-infinispan-1). The name and the number of pods will be needed to change according to infinispan cr spec. Do oc get infinispan -o yaml | grep replicas to get the replicas, oc get pod to list the pods and oc get pvc to get the persistence claims.

5. Create the new temporary PVC to use as a pivot for the data:

   $ oc set volume deployment/sleep --add -t pvc --name=pivot-data-volume-example-infinispan-0 \
   --claim-name=pivot-data-volume-example-infinispan-0 \
   --mount-path=/pivot-data-volume-example-infinispan-0 \
   --claim-mode=ReadWriteOnce \
   --claim-size=1Gi
   $ oc set volume deployment/sleep --add -t pvc --name=pivot-data-volume-example-infinispan-1 \
   --claim-name=pivot-data-volume-example-infinispan-1 \
   --mount-path=/pivot-data-volume-example-infinispan-1 \
   --claim-mode=ReadWriteOnce \
   --claim-size=1Gi
   

   ^ Above is done for volumes - oc get pvc to get the persistence claims.

6. Scale to 1 the temporary deployment and sync the data between the original PVCs and temporary PVCs:

   $ oc scale --replicas=1 deployment/sleep
   $ oc rsh deployment/sleep
   (pod) $ rsync -avxHAX --progress /data-volume-example-infinispan-0/* /pivot-data-volume-example-infinispan-0
   (pod) $ rsync -avxHAX --progress /data-volume-example-infinispan-1/* /pivot-data-volume-example-infinispan-1
   (pod) $ exit
   $ oc scale --replicas=0 deployment/sleep
   

   ^ Above is done for two volumes - oc get pvc to get the persistence claims.

7. Remove the mounted volumes and delete the original PVCs:

   $ oc set volume deployment sleep --remove --name=data-volume-example-infinispan-0
   $ oc set volume deployment sleep --remove --name=data-volume-example-infinispan-1
   $ oc delete pvc data-volume-example-infinispan-0
   $ oc delete pvc data-volume-example-infinispan-1
   

   ^ Above is done for two volumes - oc get pvc to get the persistence claims.

8. Create new PVCs in the desired storage class with the same name and the same size as the original PVCs:

   $ oc set volume deployment/sleep --add -t pvc --name=data-volume-example-infinispan-0 \
   --claim-name=data-volume-example-infinispan-0 \
   --mount-path=/data-volume-example-infinispan-0 \
   --claim-mode=ReadWriteOnce \
   --claim-size=1Gi \
   --claim-class="gp2-csi" 
   $ oc set volume deployment/sleep --add -t pvc --name=data-volume-example-infinispan-1 \
   --claim-name=data-volume-example-infinispan-1 \
   --mount-path=/data-volume-example-infinispan-1 \
   --claim-mode=ReadWriteOnce \
   --claim-size=1Gi \
   --claim-class="gp2-csi"
   

9. Do the sync of the PVC pivot to the new PVC in the desired Storage Class:

   $ oc scale --replicas=1 deploy/sleep
   $ oc rsh deployment/sleep
   (pod) $ rsync -avxHAX --progress /pivot-data-volume-example-infinispan-0/* /data-volume-example-infinispan-0
   (pod) $ rsync -avxHAX --progress /pivot-data-volume-example-infinispan-1/* /data-volume-example-infinispan-1
   (pod) $ exit
   $ oc scale --replicas=0 deployment/sleep
   

10. Remove all the mount points of the temporary deployment:

    $ oc set volume deployment sleep --remove --name=data-volume-example-infinispan-0
    $ oc set volume deployment sleep --remove --name=data-volume-example-infinispan-1
    $ oc set volume deployment sleep --remove --name=pivot-data-volume-example-infinispan-0
    $ oc set volume deployment sleep --remove --name=pivot-data-volume-example-infinispan-1
    

11. Scale the Infinispan cluster to the original value. Remember to replace example-infinispan with the correct name:

    $ oc patch infinispan example-infinispan --patch '{"spec":{"replicas":2}}' --type=merge
    

    ^ Above scales up example-infinispan infinispan cr - oc get infinispan to get the infinispan cr.

Root Cause

Data Grid Operator will create one statefulset (with the PVC) and two deployments (config-listener and operator manager). The procedure above copies the data from the pods' pvc to a temporary location, deletes the original pvc, and then re-creates it with the desired data.
The procedure works because if a PVC with the correct name (data-volume+infinispan_cr) already exists the StatefulSet that handles the pods just reuses it. The DG Operator 8.2.x allows storage class configuration, via Spec.StorageClassName.

Recovering from stale data

In case stale data ends up happening and the DG pods cannot start (crash loop) because there is still data a manual cleaning process will still be required, for that see the solution DG 8 recovering from Stale State inside pod.

Related solution:

Reading DG 8 PV Data.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.