Can you customize Metaspace and MaxMetaspace in DG as JVM argument in DG Operator?

Environment

• Red Hat Data Grid
  • Operator 8.5.x (and lower)
• Red Hat Openshift Container Platform
  • 4.x

Issue

• Can you customize Metaspace and MaxMetaspace in DG as JVM argument in DG Operator?
• Can you customize Metaspace in DG Operator?
• Can you customize MaxMetaspace in DG Operator?

Resolution

No, at the current implementation: DG Operator CSV 8.5.x (up to this versions) it is not possible, not even via the approaches listed in Customizing resource on DG Operator pod itself in OCP 4 this was request on This content is not included.JDG-6080 - still open.
There is no using extraJvmOpts option as well - it will fail:

$ oc get infinispan example-infinispan -o yaml
apiVersion: infinispan.org/v1
kind: Infinispan
...
spec:
  configListener:
    enabled: true
    logging:
      level: debug
  container:gb.
    extraJvmOpts: -XX:Metaspace=500M -XX:MaxMetaspace=500M <--- ignored (overwritten by the default)

Default values are: -XX:MetaspaceSize=32m -XX:MaxMetaspaceSize=96m.

workaround

One can configure it using the JAVA_GC_METASPACE_SIZE and JAVA_GC_MAX_METASPACE_SIZE variables directly in the OLM Subscription using the ADDITIONAL_VARS. The default values are 32m and 96m, respectively.
The only problem with this workaround is that

For details see the solution Customizing resource on DG Operator pod itself in OCP 4. In a simplified manner, for customizing the Operator itself (the pod operator) the values must be set on the OLM subscription - simplied manner:

kind: Subscription
spec:
  ...
  config:
    env:
    - name: ADDITIONAL_VARS
      value: "[\"JAVA_GC_MAX_METASPACE_SIZE\"]"
    - name: JAVA_GC_MAX_METASPACE_SIZE
      value: "300m"

The argument above MUST always be set with m (for megabyte) at the end, otherwise kilobytes will be used instead.

Verification:

$ oc rsh example-infinispan-0
sh-5.1$ jcmd
164 org.infinispan.server.loader.Loader org.infinispan.server.Bootstrap --bind-address=0.0.0.0 -l /opt/infinispan/server/conf/operator/log4j.xml -c operator/infinispan-base.xml -c operator/infinispan-admin.xml
sh-5.1$ jcmd 164 VM.info
164:
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-21.0.8.0.9-1) (21.0.8+9) (build 21.0.8+9-LTS)
---------------  S U M M A R Y ------------

Command Line: ... -XX:+ExitOnOutOfMemoryError -XX:MetaspaceSize=32m -XX:MaxMetaspaceSize=300m <----

Root Cause

The default values come from the DG Image and are nto ovewritten by the DG 8 Infinispan Custom Resource (spec's container extraJvmOpts).
The feature to add this inside the DataGrid Custom Resource is expected to be added on Data Grid Operator 8.6.x or later.
Finally, for JVM customization details see the solution Guidelines for customizing JVM flags in Data Grid 8 images.

Increasing the size of the container won't increase the Metaspace size and therefore won't avoid an OutOfMemory Exception Metaspace - if it was happening.

Diagnostic Steps

Default values:

$ oc rsh example-infinispan-0
...
sh-5.1$ jcmd 164 VM.info
164:
#
# JRE version: OpenJDK Runtime Environment (Red_Hat-21.0.8.0.9-1) (21.0.8+9) (build 21.0.8+9-LTS)

---------------  S U M M A R Y ------------

Command Line: --add-exports=java.naming/com.sun.jndi.ldap=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED -Xlog:gc*:file=/opt/infinispan/server/log/gc.log:time,uptimemillis:filecount=5,filesize=3M -Xmx512m -XX:+ExitOnOutOfMemoryError -XX:MetaspaceSize=32m -XX:MaxMetaspaceSize=96m

Custom value;

sh-5.1$ jcmd 164 VM.info
164: # JRE version: OpenJDK Runtime Environment (Red_Hat-21.0.8.0.9-1) (21.0.8+9) (build 21.0.8+9-LTS)
---------------  S U M M A R Y ------------
Command Line: -XX:MetaspaceSize=32m -XX:MaxMetaspaceSize=300m

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.