How to manually reclaim and reuse OpenShift Persistent volumes that are "Released"

Environment

• Red Hat OpenShift Container Platform (OCP)

Issue

• After deleting a PersistentVolumeClaim (PVC), the PersistentVolume (PV) is in status Released and can't be used be any other PVC.

• How to manually reclaim and reuse a PV in OpenShift Container Platform ?

Resolution

• Identify PV that is in Released state

  $ oc get pv
  NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM       STORAGECLASS   REASON    AGE
  test-pv     10Gi       RWO            Retain           Released   test-project/test-pvc                41d
  

• Inspect PV details

  $ oc get pv test-pv -o yaml
  apiVersion: v1
  kind: PersistentVolume
  metadata:
    annotations:
      pv.kubernetes.io/bound-by-controller: "yes"
    creationTimestamp: 2019-10-30T21:39:52Z
    finalizers:
    - kubernetes.io/pv-protection
    name: test-pv
    resourceVersion: "4051380"
    selfLink: /api/v1/persistentvolumes/test-pv
    uid: cea53155-fb5d-11e9-acd8-566faa560017
  spec:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    claimRef:
      apiVersion: v1
      kind: PersistentVolumeClaim
      name: test-pvc
      namespace: test-project
      resourceVersion: "6154"
      uid: 7a8823ee-fb5e-11e9-acd8-566faa560017
    nfs:
      path: /var/share/big/test-pv
      server: nfs.lab.example.com
    persistentVolumeReclaimPolicy: Retain
  status:
   phase: Released
  

• Make absolutely sure that you have a working backup of the data residing on that PV (if needed) before wiping the data

• With this example NFS share, mount the share, delete the data, unmount the share

• Remove the /spec/claimRef section with oc patch

  $ oc patch pv test-pv --type json -p '[{ "op": "remove", "path": "/spec/claimRef" }]'
  

• Check that PV state is Available afterwards

  oc get pv  
  NAME                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                                 STORAGECLASS   REASON    AGE
  test-pv                10Gi               RWO                          Retain                           Available  
  

Root Cause

• When you are finished with a volume, you can delete the PVC object from the API, which allows reclamation of the resource. The volume is considered released when the claim is deleted, but it is not yet available for another claim. The previous claimant’s data remains on the volume and must be handled according to policy.

• Dynamically provisioned volumes usually have a default ReclaimPolicy value of Delete. Manually provisioned volumes have a default ReclaimPolicy value of Retain.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.