Memory leak in EAP 6 JMX remoting

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP) 6.0

Issue

  • Memory leak when requesting attributes using JMXConnectorFactory over remoting-jmx via management interface
  • Repeated remote request of JMX attributes, as performed by a jmxquery plugin http://code.google.com/p/jmxquery/ causes heap space to grow to the limit and eventually the server to run out of memory. This is reproducible by executing the following command in a bash loop

java -cp ./jmxquery-1.2beta.jar:$JBOSS_HOME/bin/client/jboss-client.jar jmxquery.JMXQuery -U 'service:jmx:remoting-jmx://localhost:9999' -O java.lang:type=Memory -A HeapMemoryUsage -K used

Over time it is evident that memory is accumulated by thousands of instances of the following classes referenced from one instance of "java.util.HashMap$Entry[]", loaded by <system class loader>

 org.jboss.remoting3.remote.RemoteConnection
 org.jboss.remoting3.remote.RemoteControlChannel
 org.jboss.remoting3.remote.RemoteControlHandler

Resolution

Upgrade to the later release than EAP 6.1.0 coming with remoting-jxm 1.1.0.Final which incorporates fixes for Content from issues.jboss.org is not included.REMJMX-45 and Content from issues.jboss.org is not included.REMJMX-57. Of course, upgrading to the latest EAP 6 release is highly recommend.

Root Cause

Diagnostic Steps

Stacktrace when making jmx call
Here is the call stack when the plugin makes a jmx call:

"Remoting "macXYZ:MANAGEMENT" task-2@98" prio=5 tid=0xb1 nid=NA runnable java.lang.Thread.State: RUNNABLE
    at org.jboss.remotingjmx.RemotingConnectorServer.connectionOpened(RemotingConnectorServer.java:154)
    at org.jboss.remotingjmx.protocol.v1.ServerProxy.start(ServerProxy.java:167)
    at org.jboss.remotingjmx.protocol.v1.VersionOne.getProxy(VersionOne.java:55)
    at org.jboss.remotingjmx.protocol.Versions.getVersionedProxy(Versions.java:66)
    at org.jboss.remotingjmx.RemotingConnectorServer$ClientVersionReceiver.handleMessage(RemotingConnectorServer.java:244)
    at org.jboss.remoting3.remote.RemoteConnectionChannel$5.run(RemoteConnectionChannel.java:435)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

Heap Dump
Heap dumps show cumulative instances of the following objects filling up the heap

 org.jboss.remoting3.remote.RemoteConnection
 org.jboss.remoting3.remote.RemoteControlChannel
 org.jboss.remoting3.remote.RemoteControlHandler

With the instances loaded by org.jboss.modules.ModuleClassLoader and instances being referenced by one instance of java.util.HashMap$Entry[] loaded by the <system class loader>

Using the Eclipse Memory Analyzer Tool (MAT) , a heap dump illustrating the problem has the following leak suspects.

Problem Suspect 1

8,115 instances of "org.jboss.remoting3.remote.RemoteConnection", loaded by "org.jboss.modules.ModuleClassLoader @ 0xae91a7a8" occupy 144,057,480 (33.06%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>"

Keywords
org.jboss.remoting3.remote.RemoteConnection
java.util.HashMap$Entry[]
org.jboss.modules.ModuleClassLoader @ 0xae91a7a8
Details »

Problem Suspect 2

8,115 instances of "org.jboss.remoting3.remote.RemoteConnectionChannel", loaded by "org.jboss.modules.ModuleClassLoader @ 0xae91a7a8" occupy 106,338,960 (24.40%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>"

Keywords
java.util.HashMap$Entry[]
org.jboss.remoting3.remote.RemoteConnectionChannel
org.jboss.modules.ModuleClassLoader @ 0xae91a7a8
Details »

Problem Suspect 3

8,115 instances of "org.jboss.remoting3.remote.RemoteConnectionHandler", loaded by "org.jboss.modules.ModuleClassLoader @ 0xae91a7a8" occupy 83,162,520 (19.08%) bytes. These instances are referenced from one instance of "java.util.HashMap$Entry[]", loaded by "<system class loader>"

Keywords
org.jboss.remoting3.remote.RemoteConnectionHandler
java.util.HashMap$Entry[]
org.jboss.modules.ModuleClassLoader @ 0xae91a7a8
Details »
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.