Can't change Storage Solution for Quay pods

Environment

• Red Hat Quay
  • 3.x
• Red Hat OpenShift Container Platform
  • 4.x

Issue

• There are 2 PVCs present in the Quay namespace and Quay pod is binding to undesirable PVC instead of expected one :
  • local storage PVC : undesired
  • quay storage PVC : desired
• Manually changing the PVC to desired one is getting reverted to same undesirable one automatically.
• Is this expected behaviour and how can one change the PVC for Quay pods and make sure the changes persist?

Resolution

• Change the default storageclass to one that controls the desired PVC. Once done, Quay operator by default will refer to the PVC created by default storageclass.

• Steps to change the default storageclass can be found here

Root Cause

• Local PVC is getting selected over quay PVC because the storageclass used by local PVC is default storageclass. And Quay operator reverts any changes made to PVC setting back to default.

Diagnostic Steps

• Check the PVC consumed by Quay pod using below command:

$ oc describe pod quayecosystem-quay -oyaml 
...
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
  storageClassName: thin
  volumeMode: Filesystem
  volumeName: pvc-d27c46e0-eafc-4b98-91d2-4a47d3ed6ea8
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 100Gi
  phase: Bound
...

• Check all PVCs present in the Quay namespace using below command:

$ oc get pvc
NAME                                   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
example-quayecosystem-local             Bound    pvc-d27c46e0-eafc-4b98-91d2-4a47d3ed6ea8   100Gi      RWO            thin           353d
example-quayecosystem                   Bound    quay-app                                   100Gi      RWX     gp2 297d
example-quayecosystem-quay-postgresql   Bound    pvc-a1ed71f8-43bb-427b-9b3b-77bfaf494524   20Gi       RWO            thin           353d

$ oc get pvc example-quayecosystem-local -oyaml
...
kind: PersistentVolumeClaim
metadata:
  name: example-quayecosystem-local
  namespace: quay-enterprise
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  storageClassName: thin
  volumeMode: Filesystem
  volumeName: pvc-d27c46e0-eafc-4b98-91d2-4a47d3ed6ea8
...

• Check PV information related to above PVC using below command:

$ oc get pv | grep -i quay
pvc-a1ed71f8-43bb-427b-9b3b-77bfaf494524   20Gi       RWO            Retain           Bound    quay-enterprise/example-quayecosystem-quay-postgresql                thin                        381d
pvc-d27c46e0-eafc-4b98-91d2-4a47d3ed6ea8   200Gi      RWO            Retain           Bound    quay-enterprise/example-quayecosystem-local                          thin                        381d
quay-app                                   100Gi      RWX            Retain           Bound    quay-enterprise/example-quayecosystem                                gp2                         324d

$ oc get pv pvc-d27c46e0-eafc-4b98-91d2-4a47d3ed6ea8 -oyaml
...
apiVersion: v1
kind: PersistentVolume
  name: pvc-d27c46e0-eafc-4b98-91d2-4a47d3ed6ea8
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 200Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: example-quayecosystem-local
    namespace: quay-enterprise
    uid: d27c46e0-eafc-4b98-91d2-4a47d3ed6ea8
  persistentVolumeReclaimPolicy: Retain
  storageClassName: thin
  volumeMode: Filesystem
  phase: Bound
...

• Check details of storageclass named thin that is used by local PVC:

$ oc get storageclass gp2 -oyaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true" <------------------ default storage class
  name: gp2
  ownerReferences:
apiVersion: v1
kind: clusteroperator
name: storage
uid: 1069755f-b495-4e4c-a1f9-d8008ac77750
selfLink: /apis/storage.k8s.io/v1/storageclasses/thin
uid: d0a85673-a638-4ea7-8af4-bf1d9de89d61
parameters:
diskformat: thin
provisioner: kubernetes.io/vsphere-volume
reclaimPolicy: Delete
volumeBindingMode: Immediate

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.