JBoss EAP leaks 'Remoting "management-client" task' threads

Environment

• JBoss Enterprise Application Platform (EAP) 7.x

Issue

• Over time, we see a growing amount of threads like below:

"Remoting "management-client" task-9" #10439533 prio=5 os_prio=0 tid=0x00007fe6bc25c800 nid=0x1d692 waiting on condition [0x00007fe6101c0000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000074f5dc970> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Resolution

• Ensure any ModelControllerClient is properly closed when done with:

org.jboss.as.controller.client.ModelControllerClient.close();

Root Cause

• These threads are idle in an executor pool waiting for a task until the executor is shutdown. The executor belongs to remoting/CLI objects created through the CLI API so these executor threads will leak if a CLI ModelControllerClient is not properly closed when done with.

Diagnostic Steps

• Capture a heap dump and inspect to see any application code reference to the associated remoting/CLI objects

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.