Failed to Configure NFS Share for image-registry Storage

Solution Unverified - Updated

Environment

UPI install on vSphere of Red Hat OpenShift Container Platform v4.x

Issue

When attempting to utilize an NFS share for image-registry storage, the PVC created by the image registry operator fails with an error such as:

Failed to provision volume with StorageClass "thin": invalid AccessModes [ReadWriteMany]: only AccessModes [ReadWriteOnce] are supported

Resolution

To work around this issue, perform the following actions:

1. Delete the PV and PVC

timeout 2 oc delete pv pv-name
timeout 2 oc delete pvc image-registry-storage -n openshift-image-registry

2. Delete image-registry pods

oc get pods -n openshift-image-registry

# for each pod with a name beginning with image-registry- , run:
timeout 2 oc delete pod image-registry-some-random-numbers -n openshift-image-registry

3. Create a new storage class which does not have a configured access mode.

oc create -f - <<EOF
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: non-dynamic 
provisioner: no-provisioning 
parameters:
EOF

4. Disable the "thin" storage class as the default storageclass

oc patch storageclass thin -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "false"}}}'

5. Enable the "non-dynamic" storage class as the default storageclass

oc patch storageclass non-dynamic -p '{"metadata": {"annotations": {"storageclass.kubernetes.io/is-default-class": "true"}}}'

6. Create a new PV which references the new storage class.

oc create -f - <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0001
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteMany
  nfs:
    path: /server
    server: 10.x.x.138
  persistentVolumeReclaimPolicy: Retain
  storageClassName: non-dynamic
EOF

7. Create a new image-registry PVC which references the custom storage class.

oc create -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  finalizers:
  - kubernetes.io/pvc-protection
  name: image-registry-storage
  namespace: openshift-image-registry
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 100Gi
EOF

Root Cause

On a vSphere cluster, the default storage class is thin which has an access mode of ReadWriteOnly. The PVC created by the image registry does not define a storage class and as a result, the default storage class is used. The image registry PVC is configured to have an access mode of ReadWriteMany.

Diagnostic Steps

Check the state of the image registry PVC by running:

oc describe pvc -A

The image-registry-storage PVC should be in a pending state with an output similar to the following.

Name:          image-registry-storage
Namespace:     openshift-image-registry
StorageClass:  thin
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/vsphere-volume
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Events:
  Type       Reason              Age                  From                         Message
  ----       ------              ----                 ----                         -------
  Warning    ProvisioningFailed  5s (x11 over 8m18s)  persistentvolume-controller  Failed to provision volume with StorageClass "thin": invalid AccessModes [ReadWriteMany]: only AccessModes [ReadWriteOnce] are supported
Mounted By:  image-registry-7dcbf9cd8b-xdx6l
             image-registry-89b9688bd-h54sz
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.