Data Grid 8 OpenSSL session objects native accumulation

Solution Verified - Updated

Environment

  • Red Hat Data Grid (RHDG)
    • 8.2
    • 8.3

Issue

  • RHDG 8 native consumption continuous to increase
  • Native Memory accumulation during a period of time after upgrade to Operator 8.3
  • RHDG pods are restarted after a while because of memory consumption

Resolution

Fix version

Data Grid 8.4, released in November 2022, uses a fixed version of the OpenSSL library (2.2.5).
Tracked by This content is not included.JDG-5745

Workaround for 8.2/8.3

Disable OpenSSL using by adding -Dorg.infinispan.openssl=false to the server start command - it will make able to use native Java SSL instead of OpenSSL, which might be slower than openssl.

Root Cause

This is related to OpenSSL session objects, related to This content is not included.WFSSL-51 and This content is not included.WFSSL-56. Given that RHDG 8.3.1 ships with WildFly OpenSSL natives 2.1.0.SP01 and the fix needs 2.1.1.Final - which is used by RHDG 8.4.

In regards to SSL/TLS connection in OpehShift environment, the handling is done by Data Grid. As per documentation the Route is set to Passthrough. Therefore, the TLS connection is done between DG client and server. OCP routers just let the SSL connection pass, reference.

Diagnostic Steps

Verify the heap dump, where there will be many OpenSSLSessions opened - however, native accumulation (given not in Direct Buffers) cannot directly be seen on the heap dumps.
Or VM.info can be used to compare RSS utilization:

Comparing to RSS - before disabling:

$ grep -ri "Resident"
example/VM.info.txt:Resident Set Size: 1_598_116K (peak: 1598200K) (anon: 1548496K, file: 49568K, shmem: 52K) <--- anon with 1.5gb

After disabling:

$ grep -ri "Resident"
example/VM.info.txt:Resident Set Size: 1_029_336K (peak: 1_184_572K) (anon: 998544K, file: 30792K, shmem: 0K) <--- anon with 998 mb
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.