Metrics fails to deploy with Persistent Storage when PVC is self-created

Environment

• Red Hat OpenShift Enterprise 3.0-3.2
• Red Hat OpenShift Container Platform 3.3+

Issue

• I created the PV and PVC for my metrics and then tried to deploy but hit problems
• Deploying metrics for the first time and I get an error about storage:

<DATE_TIME>   <DATE_TIME>   2         hawkular-cassandra-1-7voky   Pod                                                 Warning   FailedMount        {kubelet test.example.com}   Unable to mount volumes for pod "hawkular-cassandra-1-7voky_openshift-infra(96765ee3-8c28-11e6-916f-005056a6d599)": timeout expired waiting for volumes to attach/mount for pod "hawkular-cassandra-1-7voky"/"openshift-infra". list of unattached/unmounted volumes=[cassandra-data]
<DATE_TIME>   <DATE_TIME>   2         hawkular-cassandra-1-7voky   Pod                                                 Warning   FailedSync         {kubelet test.example.com}   Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "hawkular-cassandra-1-7voky"/"openshift-infra". list of unattached/unmounted volumes=[cassandra-data]
<DATE_TIME>   <DATE_TIME>   29        heapster-lszxv               Pod                     spec.containers{heapster}   Warning   Unhealthy          {kubelet test.example.com}   Readiness probe failed: The heapster process is not yet started, it is waiting for the Hawkular Metrics to start.

Resolution

• Remove the PVC that you created to be attached to the intended PV and let/help the PV become Available again

oc delete pvc <NAME>

• Then delete all of the components for metrics and redeploy

Root Cause

• The Metrics Deployer creates its own PersistentVolumeClaim when you indicate that you want to use persistent storage.
  • This means that if you have manually created your own PVC and let it bind to the PV you created for Metrics, the deployer will not be able to successfully connect the Cassandra pods to the storage, as it will already be bound.
• The default naming scheme for the PVC created by the Metrics Deployer is metrics-cassandra-# so if there is already a PVC bound to your intended Metrics PV, you will need to clear it up before it will work.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.