DG 8 Persistence Volume details and how to read PV

Environment

• Red hat OpenShift Container Platform (OCP)
  • 4.x
• Red Hat Data Grid (RHDG)
  • 8.x

Issue

• How to read PV data in OCP 4?
• Issues about reading DG 8 PV Data in OCP 4?

Resolution

FAQ about Reading DG 8 PV Data::

Q0. What is PV? What does it mean? Does PV persist data after the pod restart?
A0. PV means persisted volume. It means persisted because it persists pod restarts. Everything in $RHDG_HOME/server/data that is meant to survive pod restarts. Therefore, deleting the pod does not mean the PV gets cleaned, although type of the PV is delete, which only matters when the PV gets unbounded.

Q1. Does the user create PV on DG 8?
A1. See Root cause for the complete details on PV/PVC interaction.

Q2. When we set persistence on the cache configuration for file-storage, which writes to the PV. If the pod drops, will the PV be persisted? - baseline question.
A2. Yes, given writing to the PV (persisted) so then PV will be persisted.

Q3. How to retrieve the data from the PV? (when persistence is enabled)? If the DG goes down?
A3. With the file-store, the data is stored in binary format and there aren't any 3rd party tools to read it. You would need to mount the PVC(s) in a Pod(s) to retrieve PV access.

Q4. How to retrieve the data itself?
A4. When the DG is restarted, the PV is mounted with the data and DG is able to read the data again.

For example, if you want to just recreate a cluster using the persistent data: if the PVCs remain, it's possible to create a new Infinispan CR with the same name as before and the same PVCs will be re-used. However, if you just want to extract the raw files for future use/debugging then you need to:

1. mount the PVC to a pod
2. and extract the data from the location you mount it using oc cp ... for extraction.

See Migrate persistent data to another Storage Class in DG 8 Operator in OCP 4.

Q5. What is the format the data is stored? Is it ascii?
A5. Binary format for all cache store configurations except the SQL store, where is field is mapped to a table column for the SQL DB. However, nothing is stored in ascii though, nor human readable format.

Q6. Given that after eviction, the data should be on the file store, if passivation has been configured. How is the data persisted?
A6. It depends on the configuration set: eviction, expiration, and passivation. On Dg 8 Operator the configuration will be on the respect Cache Custom Resources - Not Infinispan's CR.

See details on Passivation impact on Storage size/Memory size in DG Operator in Openshift 4.

Q7. Can one read the data with a different party from the RHDG store?
A7. This is not supported or recommended. Only a SQL store or a custom-store could be used, where a custom-store is complicated and we would not recommend to use this.
Since RHDG 8 is the owner of the data, so tunneling it outside with a third party need extra considerations, including simultaneous access on data.

Q8. What about NFS? And what about NFS vendor recommendation
A8. From DG 8 - file store persistence NFS is not recommended.
Red Hat does not have a recommendation of vendor specifically, nor it can provide one. The customers need to assess them according to their needs. NFS is usually not recommend because it does not implement POSIX, which leads to potential issues (locking being the most important). Other products may not recommend NFS as well for this same reason.

Q9. What StorageClass will be used if nothing is specified?
A9. The default StorageClass of the cluster should be used if none is specified on the Infinispan CR.

Q10. What would be the procedure to migrate an OCP PV's without losing data for the DG cluster
A10. The simplest procedure is detailed here: Migrate persistent data to another Storage Class in DG 8 Operator in OCP 4 - The procedure works because if a PVC with the correct name (data-volume+infinispan_cr) already exists the StatefulSet, which handles the pods, will just reuses it.

Q11. Can the user increase the PV or PVC sizes?
A11. When using DG Operator, the user can set/increase the default storage size the PVC spec not the PV directly on the Infinispan Cluster custom resource. The user cannot increase the size of the PV directly from the custom resource - or on-the-fly for example. But see details below for manual provisioning (in comparison to dynamic provisioning). For several reasons, including that not all PVs support resizing, the spec storage field cannot be changed at runtime, so a new needs to be created.

Q12. Does changing the default SA/SCC in DG Operator deployment impacts the deployment?
A12. Yes it does impact. If deployment doesn't set the service account, so then pod get the default SA and the SCC used will be restricted(-v2), for security reasons.
In case the default SCC is set for anyuid via the default SA with the role binding, so then this causes filesystem permissions errors in the pod. Changing the SCC for anyuid will impact directly the deployment.

Q13. How to clean stale data inside the PV?
A13. There are a few options, explained on the solution DG 8 recovering from Stale Sate inside its pod:

• Set the number of replicas on Infinispan Custom Resource to zero, then delete manually the PVC. A new PVC will come up fully clean.
• Set the number of replicas on Infinispan Custom Resource to zero, then Mount PVC to another pod and then clear the data accessing directly, this procedure can be used for access and migration as well.
• Later versions of the DG Operator are able to delete/recreate the PVC directly just by setting the Infinispan CR to zero, following This content is not included.Operator Stale file-store data possible after cluster scale. However, only on the second pod forward in a cluster.

Q14. PV type details and how it behaves?
A14. The PV will be bounded to each DG Operand pod (and DG Operator - on later versoins). Access mode: ReadWriteOnce (RWO) with Reclaim policy for Delete, which means: when the PV gets unbounded (PVC gets deleted), so the PV will get deleted:

$ oc get pv
NAME CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
pvc-id   1Gi        RWO            Delete           Bound    datagrid-operator/data-volume-CR-infinispan-1   managed-csi    <unset>                          
pvc-id   1Gi        RWO            Delete           Bound    datagrid-operator/data-volume-CR-infinispan-2   managed-csi    <unset>                          
pvc-id   1Gi        RWO            Delete           Bound    datagrid-operator/data-volume-CR-infinispan-0   managed-csi    <unset>  
...
$ oc get pvc
NAME               STATUS   VOLUME CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS
data-volume-CR-0   Bound    pvc-id   1Gi        RWO            managed-csi    <unset>                 
data-volume-CR-1   Bound    pvc-id   1Gi        RWO            managed-csi    <unset>                 
data-volume-CR-2   Bound    pvc-id   1Gi        RWO            managed-csi    <unset>                                         

Root Cause

PV vs PVC deep explanation

The operator never deals with PV directly, in the same way we use the Pod abstraction instead of interacting with Nodes directly.
The operator provides a PVC spec as part of the cluster StatefulSet. The StatefulSet controller then creates a PVC for each of the Pods in the cluster, using the configured amount of storage.
For details on PV manual provisioning see Deploy Red Hat Data Grid 8 in OpenShift 4 using Operator.In summary:

• For dynamic provisioning: It is the default method used. However, on the Infinispan CR user can set a custom storageClassName. The persistent volume claim uses the storage class that has the storageclass.kubernetes.io/is-default-class annotation set to true.
• For manual provisioning: the user creates a dg8storageclass and then the PV, which would then be picked up by the DG pods accordingly - which creates the claims. See solution Deploy Red Hat Data Grid 8 in OpenShift 4 using Operator.

Finally, the pods deployed by the DG Operator have RWO set for the PVC, so the DG pods need to be set to zero (on Infinispan CR set to zero), before injecting from another.

It's possible for the user to set the specific the amount of storage allocated to each PVC by providing a value for spec.service.container.storage in the Infinispan Custom Resource, larger than the default 1Gi.
See more details on allocating storage class services.

Issues

DG 8.3 has issues with persistence for FileStore as explained on the solution How an unshared CacheStore behave in a RHDG clustered environment? - preloading doesn't solve this issue. The only workaround is setting purge=true (or not using a persistent PV) that would stop the issue. However, the main implication is that the data will not survive a full cluster restart.

DG Operator will use the default Storage Class, i.e. the class with storageclass.kubernetes.io/is-default-class. To change the default storage, change that annotation therefore.

Diagnostic Steps

1. See pod logs for corruption of file:

16:18:24,543 ERROR (non-blocking-thread--p2-t1)
[org.infinispan.server.hotrod.BaseRequestProcessor] ISPN005003: Exception reported
org.infinispan.persistence.spi.PersistenceExcep8on: ISPN029019: Cannot load key
WrappedByteArray[03\>0A\5\0\0\0\0\2\5\6\9\7 (13 bytes)] from index.
at
org.infinispan.persistence.sifs.NonBlockingSoCIndexFileStore.lambda$load$17(NonBlockingSoCInde
xFileStore.java:605)
at
java.base/java.u8l.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:3
5)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982)
at
org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1
486)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.IllegalStateExcep8on: Error reading from 25:16511130
Page 4 of 7
at org.infinispan.persistence.sifs.EntryRecord.readEntryHeader(EntryRecord.java:93)
at org.infinispan.persistence.sifs.IndexNode$LeafNode.getHeaderAndKey(IndexNode.java:1033)
at org.infinispan.persistence.sifs.IndexNode$LeafNode.loadRecord(IndexNode.java:1063)
at org.infinispan.persistence.sifs.IndexNode$ReadOpera8on$1.apply(IndexNode.java:224)
at org.infinispan.persistence.sifs.IndexNode$ReadOpera8on$1.apply(IndexNode.java:221)
at org.infinispan.persistence.sifs.IndexNode.applyOnLeaf(IndexNode.java:295)
at org.infinispan.persistence.sifs.Index.getRecord(Index.java:120)
at org.infinispan.persistence.sifs.Index.getRecord(Index.java:106)
at
org.infinispan.persistence.sifs.NonBlockingSoCIndexFileStore.lambda$load$17(NonBlockingSoCInde
xFileStore.java:596)
... 6 mo

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.