Generating and analyzing JFR in Data Grid 8 in OCP

Environment

• Red hat OpenShift Container Platform (OCP)
  • 4.x
• Red Hat Data Grid (RHDG)
  • 8.x
  • Operator

Issue

How to create a JFR in Data Grid 8 in OCP?
How to analyze the JFR created in Data Grid 8 in OCP?

Resolution

Below described the process without jcmd:

1. Set StartFlightRecording in spec.extraJvmOpts:

    extraJvmOpts: '-Xlog:gc*=info:file=/tmp/gc.log:time,level,tags,uptimemillis:filecount=10,filesize=1m  -XX:StartFlightRecording=duration=200s,filename=flight.jfr'
   

2. The file flight.jfr will be on /opt/infinispan:

   oc rsh $podname
   sh-4.4$ ls
   Copyright.txt  README.adoc  bin  boot  diag  docs  flight.jfr <-- lib  server  static  version.txt
   

3. Fetch the file from pod to local: oc cp $podname:/opt/infinispan/flight.jfr .

   oc cp dg-cluster-jfr-0:/opt/infinispan/flight.jfr .
   

4. To analyze it, open in Java Mission Control:

./jmc <-- open the file

Example:

Cryostat

Currently DG Operator 8.3.x does not have a seamless integration with JMX.

Root Cause

As explained on solution How to use JDK Flight Recorder (JFR) one can start a recording of JFR file via jcmd or set the JVM to start already with it via StartFlightRecording. The example above describe StartFlightRecording usage.
For alternatives to create a heap dump, see Alternatives for creating heap dump in a DG 8 even without the JDK](https://access.redhat.com/solutions/6964022).

Options for collecting JFR files

Adequate situations for using JFR files

• Investigation of heap memory (leaks, usage)
• Thread usage investigation
• class/object investigation

Not adequate situations for using JFR files

• if JMX is not enabled by default and cannot use the flag StartFlightRecording
• Using JFR one can can see how much physical memory is being used, however not what is using it in native/outside of java.
• Force a Full GCs (used for Full heap investigations)
• To collect and show thread dumps on the spot (one first collect JFR file, then sees it)

For those scenarios above:
Use jcmd command see Alternatives for creating heap dump in a DG 8 even without the JDK in case the image does not have jcmd binary.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.