Editing Encryption at Infinispan Custom Resource on the fly leads to misbehavior

Solution Verified - Updated

Environment

  • Openshift Container Platform
    • 4.x
  • Data Grid
    • 8.x
    • Operator

Issue

  • Changing the Infinispan Custom Resource from Service to None breaks the Route.
  • Changing the Infinispan Custom Resource from Service to Secret breaks the Route.
  • Changing the Infinispan Custom Resource encryption led to misbehavior

Resolution

As explained on Data Grid Operator Encryption types and implications, there are three types of Encryption and how they act:

TypeEncryption fileRoute detailDG pod detail
ServiceThe encryption is done by the Service-CA Operator inside the CR's serviceRoute will have passthrough terminationCertificate comes from Service-CA Operator and injected by the DG Operator
SecretThe encryption is done by a encrypt file injected via secreatRoute will have passthrough terminationCertificate comes from secret created by user.
NoneThe encryption is done by the Service-CA Operator inside the CR's serviceRoute will NOT have passthrough terminationNo certificate

Changing the Encryption's type on-the-fly, can lead to misbehavior exactly because of the above. The recommendation is to create the Infinispan Custom Resource with a certain type and stick with it, and if need to change it follow as below:

  • Understand it can misbehave and how it operates
  • Separate a windows for that change
  • If misbehave, be prepare to re-create the Infinispan Custom Resource

Root Cause

When you create a DG Infinispan with a certain type of Encryption and that has implications on the DG pods and the Route. Exemplification scenarios:

  • Secret || Service -> None: Let's say from Secret || Service to None, the Route will still have the termination, passthrough. However, when a user sets the Infinispan Custom Resource to Encryption None, then the route will be access without and that's where the problem lies.

  • None -> Secret || Service: Let's say from None to Service|Secret. The Route needs to be re-created because the None's Route doesn't have termination. It will misbehave if the route does not have termination and the backend pods have.

  • Service -> Secret: Let's say from Service to Secret the Route will have a valid termination passthrough. However, the pods need to be restarted fully and properly with the secret's encryption details. The DG pods need to fully abide with the new encryption certificates, otherwise it will misbehave.

As a general rule: create the Infinispan Custom Resource with a certain encryption and stick with it and avoid change at runtime. If you delete the route as a workaround for this, the new route may take 10/15 min for be re-spawned.

Diagnostic Steps

  1. Verify the Infinispan Custom Resource encryption type (service, secret, none) vs the
  2. Verify the respective Route's service and the related endpoint.
  3. Verify the haproxy logs, if they return CD - it means the Route is mistaken and likely need to be deleted: 
2025-09-03T21:49:35.356316+00:00 router-default-66f875dcf9-2rcrf router-default-66f875dcf9-2rcrf haproxy[1387]: 76.253.191.127:55964 [03/Sep/2025:21:49:35.326] public_ssl be_tcp:dg-custom-realm:example-infinispan-no-tls-external/pod:example-infinispan-no-tls-1:example-infinispan-no-tls:infinispan:10.128.2.86:11222 4/2/29 83 CD 19/10/0/0/0
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.