How to migrate data between PVs in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4

Issue

  • How to migrate data between PVs in OpenShift 4?
  • Persistent volume (PV) is getting full.
  • Need to migrate old PV data to different storage.
  • How to extend or increase persistent volume size?
  • What is the method to swap out a persistent volume for another persistent volume?
  • OpenShift PV is out of disk or seeing disk full errors.

Resolution

NOTE: For more complicated scenarios where multiple volumes need to be migrated or where volume data must be moved across clusters, Red Hat recommends using OpenShift API for Data Protection (OADP) or the Migration Toolkit for Containers (MTC).

For more details on using OADP, please see:

For more details on using the MTC, please see:

Procedure


The following steps can be used to manually migrate data from an old source PV to a new target PV (please note that this will not work with [raw block volumes](https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/storage/understanding-persistent-storage#block-volume-support_understanding-persistent-storage)).

Manually migrating data between PVs

  1. Get the old PVC name along with its pod volume name and mount path. For example:
  $ oc get pvc
  NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
  my-app-pvc    Bound    pvc-05806210-f4a6-412d-a84c-3379fae8dd88   2Gi        RWO            gp3-csi

  /// -------------------------------------------------------------
  /// The following command simply returns a list of volumes in the
  /// app deployment; it does **not** change the deployment
  /// -------------------------------------------------------------

  $ oc set volume deployment my-app-name
    my-app-name
      pvc/my-app-pvc (allocated 2GiB) as my-app-volume
        mounted at /my-volume-mount

In the above example:

  • my-app-pvc is the PVC name
  • my-app-volume is the volume spec name that includes the PVC in the pod definition
  • /my-volume-mount is the path the PVC is mounted to inside the application pod
  1. Scale down the application deployment where the old PV is mounted (this prevents storage writes from occurring during the copy procedure and ensures a consistent state):
$ oc scale deployment my-app-name --replicas=0
  1. Start a new temporary pod deployment using the rhel-tools image:
$ oc create deployment migrator --image=registry.access.redhat.com/rhel7/rhel-tools -- tail -f /dev/null
  1. Mount the old PVC to the temporary migrator pod (the --claim-name argument is used to specify the PVC):
$ oc set volume deployment/migrator --add -t pvc --name=old-volume --mount-path=/old-volume --claim-name=my-app-pvc
  1. Create a new target PVC to copy the old volume data to. Use the below example PVC as a template, changing name, namespace, accessMode, requests, and storageClassName as needed.

NOTE: If dynamic provisioning is not being used, then a new static PV will need to be created manually prior to this step.

```
    $ cat new-pvc.yaml
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: new-pvc
      namespace: hello
    spec:
      accessModes:
      - ReadWriteOncePod
      resources:
        requests:
          storage: 2Gi
      storageClassName: gp3-csi
      volumeMode: Filesystem

    $ oc apply -f new-pvc.yaml
```
  1. Attach the new PVC to the temporary migrator pod:
$ oc set volume deployment/migrator --add -t pvc --name=new-volume --mount-path=/new-volume --claim-name=new-pvc
  1. rsh into the pod:
$ oc rsh deployment/migrator
  1. Migrate data to the new volume storage:
sh-4.2$ rsync -avxHAX --progress /old-volume/* /new-volume

NOTE: The above rsync command preserves metadata (permissions, ACLs, extended attributes, ...) from the files and directories copied, so the destination filesystem needs to be compatible with the origin filesystem's metadata. If the destination filesystem is not compatible, check to ensure the copied files still work with the application without the original metadata.

  1. Update the application deployment so that it uses the new volume (be sure to specify the correct mount path that was confirmed in the first step):
sh-4.2$ exit
$ oc set volume deployment my-app-name --remove --name=my-app-volume
$ oc set volume deployment my-app-name --add -t pvc --name=my-app-volume --mount-path=/my-volume-mount --claim-name=new-pvc
  1. Remove the temporary migrator pod:
$ oc delete deployment migrator
  1. Scale up the application:
$ oc scale deployment my-app-name --replicas=1

Alternative method


In OpenShift 4, the `oc run` command creates a stand-alone `pod` object with no ability to add or change the `volumes` list. However, after creating a simple pod with `oc run`, the pod `yaml` definition can be exported to a file and the `volumes` list manually edited. A new pod can then be started from the updated `yaml` definition using the `oc create -f` command. 
$ oc run migrator --image=registry.access.redhat.com/rhel7/rhel-tools -- tail -f /dev/null

$ oc get pod migrator -o yaml > temp-pod.yaml

$ vi temp-pod.yaml

{...}
spec:
  containers:
  ...
  volumeMounts:
    - mountPath: /var/lib/myclaim
      name: myclaim
volumes:
- name: myclaim
   persistentVolumeClaim:
     claimName: myclaim
{...}

$ oc delete pod migrator

$ oc create -f temp-pod.yaml

Advanced method


Alternatively, the `oc run` command with option `--overrides=` allows for more control of the pod definition during startup. 
$ oc run migrator --image=registry.access.redhat.com/rhel7/rhel-tools --overrides='{ "spec":{"containers":[{"image":"registry.access.redhat.com/rhel7/rhel-tools","name":"migrator","volumeMounts":[{"mountPath":"/var/lib/myclaim","name":"myclaim"}]}],"volumes":[{"name":"myclaim","persistentVolumeClaim":{"claimName":"myclaim"}}]}}'  -- tail -f /dev/null

Root Cause

For more complicated backup or migration scenarios, it is recommended to use OpenShift API for Data Protection (OADP) or the Migration Toolkit for Containers (MTC). For simple cases, data can be manually copied from one PV to another.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.