JBoss reaching OOME during start up after update to EAP 7.4 Update 11

Environment

• JBoss Enterprise Application Platform (EAP) 7.4 Update 11

Issue

• We updated to EAP 7.4 Update 11 and now face OOMEs in start up.
• An OOME is reached in com.redhat.insights.jars.JarUtils.computeSha:

Thread 0xf0d22c40
  at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:48)^M
  at java.util.Arrays.copyOf([BI)[B (Arrays.java:3236)^M
  at java.io.ByteArrayOutputStream.grow(I)V (ByteArrayOutputStream.java:118)^M
  at java.io.ByteArrayOutputStream.ensureCapacity(I)V (ByteArrayOutputStream.java:93)^M
  at java.io.ByteArrayOutputStream.write([BII)V (ByteArrayOutputStream.java:153)^M
  at com.redhat.insights.jars.JarUtils.computeSha(Ljava/io/InputStream;Ljava/lang/String;)Ljava/lang/String; (JarUtils.java:164)^M
  at com.redhat.insights.jars.JarUtils.computeSha(Ljava/net/URL;Ljava/lang/String;)Ljava/lang/String; (JarUtils.java:155)^M
  at com.redhat.insights.jars.JarUtils.computeSha1(Ljava/net/URL;)Ljava/lang/String; (JarUtils.java:132)^M
  at com.redhat.insights.jars.JarAnalyzer.getJarInfoSafe(Ljava/lang/String;Ljava/net/URL;)Lcom/redhat/insights/jars/JarInfo; (JarAnalyzer.java:109)^M
  at com.redhat.insights.jars.JarAnalyzer.process(Ljava/lang/String;Ljava/net/URL;)Ljava/util/Optional; (JarAnalyzer.java:74)^M
  at org.jboss.eap.insights.report.JBossInsightsDeploymentSubReport.generateReport()V (JBossInsightsDeploymentSubReport.java:109)^M
  at org.jboss.eap.insights.report.JBossInsightsWrapperSubReport.generateReport()V (JBossInsightsWrapperSubReport.java:69)^M
  at com.redhat.insights.AbstractTopLevelReportBase.generateReport(Lcom/redhat/insights/Filtering;)V (AbstractTopLevelReportBase.java:199)^M
  at com.redhat.insights.InsightsReportController.generateConnectReport()V (InsightsReportController.java:149)^M

Resolution

• Apply EAP 7.4 Update 12 or later
• Add -Drht.insights.java.opt.out=true to your JVM options or set it as a system property, which will disable the insights resource.

Root Cause

• Red Hat Insights functionality was enabled in EAP 7.4 Update 11 and its reporting behavior adds extra overhead. Insights is only enabled if EAP is running on RHEL with the Insights client installed.
• With large configurations, excessive overhead is seen from recursive read-resource operations
  • This content is not included.WFCORE-6386
    • This content is not included.JBEAP-24933
    • This content is not included.JBEAP-24951
• With large deployments, excessive overhead is seen from reading the whole jar to calculate a SHA sum
  • This content is not included.MWTELE-73
    • This content is not included.JBEAP-24973
  • This content is not included.MWTELE-76

Diagnostic Steps

• Example of overhead in heap dump:

Class Name             |   Objects | Shallow Heap |  Retained Heap
org.jboss.dmr.ModelNode| 1,700,036 |   40,800,864 | >= 136,520,392

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.