Data Grid 8 Operator pod/statefulset is stuck

Solution Verified - Updated

Environment

  • Red hat OpenShift Container Platform (OCP)
    • 4.x
  • Red Hat Data Grid (RHDG)
    • Operator 8.x <--- most of the times 8.3.x

Issue

  • Data Grid 8 Operator statefulset is stuck in the number of pods
  • Data Grid 8 Operator statefulset is stuck in another service type
  • DG 8 statefulset does not correspond to the Infinispan CR definition or the correspods but the pods do not honor the configuration

Resolution

On RHDG in OCP 4, the DG Operator deploys Data Grid instances (containers, inside pods) by listening to Infinispan CRs.
From the user specifications on the Infinispan CR (or any CR for that matter) Data Grid Operator will create the resources accordingly.
For Infinispan CR the DG Operator will create a statefulset, which deploys the DG pods - the pods are not directly deployed.
Sometimes however, for a series of reasons the pods can get stuck on a previous configuration.
Given the pods are spawned from the statefulset so then if the statefulset has the outdated configuration, deleting the pods won't be enough.

Below should be an exceptionality

Issue

Pods are stuck on previous configuration;

Recommendation

Delete the statefulset that spawns the pods and the ReplicaSet that deploy the pods.

Root Cause

Statefulset gets stuck with previous configuration, e.g. service type Cache (vs type DataGrid);
See solution Deprecated service type Cache in DG 8 in OCP 4 for how this can be caused (migration section).

Diagnostic Steps

  1. Review the Infinispan Cr
  2. Review the pods
  3. Review if the statefulset

Rationale

  • If the pods have the same configuration as the statefulset, and does not correspond to the infinispan Cr, so then the statefulset is outdated and must be deleted;

  • If the pods have the a different configuration than the statefulset (which would then correspond to the Infinispan CR), so then the ReplicaSet is outdated and must be deleted;

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.