High CPU and heap overhead in JMX when many rars are deployed

Solution Unverified - Updated 2 Aug 2024

Environment

JBoss Enterprise Application Platform (EAP) 6.x

Issue

We have many (hundreds) of rars deployed. We noticed substantial increased heap usage in this case and CPU is persistently high when we connect via JMX and mbeans are never rendered when viewed via jconsole. The heap dump shows the org.jboss.as.connector.subsystems.resourceadapters.IronJacamarResources are quite large. And thread dumps show CPU is high in calls like the following:

"RMI TCP Connection(1)-127.0.0.1" #100 daemon prio=9 os_prio=0 tid=0x00007f02240c5000 nid=0x3a43 runnable [0x00007f024d6f7000]
   java.lang.Thread.State: RUNNABLE
        at java.security.AccessController.getStackAccessControlContext(Native Method)
        at java.security.AccessController.getContext(AccessController.java:820)
        at org.jboss.as.controller.SecurityActions.getCaller(SecurityActions.java:132)
        at org.jboss.as.controller.AbstractOperationContext.getCaller(AbstractOperationContext.java:1017)
        at org.jboss.as.controller.OperationContextImpl.authorize(OperationContextImpl.java:1284)
        at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:646)
        at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:632)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.addParentResource(ReadResourceDescriptionHandler.java:786)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:774)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getAllActualResourceAddresses(ReadResourceDescriptionHandler.java:778)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.getLocalResourceAddresses(ReadResourceDescriptionHandler.java:713)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.initLocalResourceAddresses(ReadResourceDescriptionHandler.java:702)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler$ReadResourceDescriptionAccessControlContext.access$200(ReadResourceDescriptionHandler.java:689)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler.doExecuteInternal(ReadResourceDescriptionHandler.java:204)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler.doExecute(ReadResourceDescriptionHandler.java:162)
        at org.jboss.as.controller.operations.global.ReadResourceDescriptionHandler.execute(ReadResourceDescriptionHandler.java:155)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702)
        at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537)
        at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314)
        at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144)
        at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331)
        at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:188)
        at org.jboss.as.jmx.model.ResourceAccessControlUtil.getResourceAccess(ResourceAccessControlUtil.java:85)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:51)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
        at org.jboss.as.jmx.model.RootResourceIterator.iterate(RootResourceIterator.java:43)

Resolution

Apply the This content is not included.EAP 6.4.8 CP
As a workaround, remove expose-expression-model / expose-resolved-model to not expose the JBoss Management layer via JMX:

        <subsystem xmlns="urn:jboss:domain:jmx:1.3">
            <expose-expression-model/>
            <expose-resolved-model/>
            <remoting-connector/>
        </subsystem>

Root Cause

This content is not included.BZ-1309747
Content from issues.jboss.org is not included.JBEAP-3498
Content from issues.jboss.org is not included.WFLY-6250
It's evident that it is iterating over each deployment (each resource adapter counting as a deployment), but for each deployment, it is also iterating over all resource adapter's and their config properties. So it would iterate over all resource adapters once for each deployment. This causes high CPU in n^2 iterations and increased heap in n^2 references.

Diagnostic Steps

Capture a heap dump during the high CPU to confirm what resources are iterated over.

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.