DG 8 recovering from Stale State inside pod

Solution Verified - Updated

Environment

  • Red hat OpenShift Container Platform (OCP)
    • 4.x
  • Red Hat Data Grid (RHDG)
    • 8.x
    • Operator

Issue

  • How to recover from state state
  • DG Operator recovering from Stale State inside pod

Resolution

Following the full explanation on DG 8 Persistence Volume details and how to read PV, there are a few options for recovering from a state state:

  • Set the number of replicas on Infinispan Custom Resource to zero, then delete manually the PV/PVC. This is possible because the statefulset will bring a new PVC fully clean.
  • Deleting the PV data in the first pod may be required - that's because the DG Operator PV IMPLEMENTATION CHANGED and the data is not reset when the POD ZERO will be kept even coming at zero pods.
  • Set the number of replicas on Infinispan Custom Resource to zero, then Mount PVC to another pod and then clear the data accessing directly, this procedure can be used for access and migration as well, see the solution Migrate persistent data to another Storage Class in DG 8 Operator in OCP 4 - for full migration procedure.

Later versions of the DG Operator will be able to delete/recreate the PVC directly just by setting the Infinispan CR to zero, following Operator Stale file-store data possible after cluster scale. But the cleaning only will happen on the second pod onwards.

The same can be done in Helm charts deployment, where the user sets the replicas on the statefulset instead of the custom resources (there are no custom resources in helm charts).

Root Cause

Upgrading/Re-installing the DG Operator won't purge the data - so won't solve this problem necessarily

For full explanations on PV see DG 8 Persistence Volume details and how to read PV, there are a few options for recovering from a state state discussed above.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.