JBoss EAP has many threads named like "pool-161-thread-20"

Environment

• JBoss Enterprise Application Platform (EAP)
  • 5.x
  • 6.x
  • 7.x

Issue

• There are hundreds of threads all looking like this:

"pool-132-thread-6" prio=10 tid=0x00007f7630232800 nid=0x1276 waiting on condition [0x00007f76453a6000]  
   java.lang.Thread.State: WAITING (parking)  
    at sun.misc.Unsafe.park(Native Method)  
    - parking to wait for  <0x00000007afba0f90> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)  
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)  
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)  
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)  
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)  
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)  
    at java.lang.Thread.run(Thread.java:662)

• The names will vary: "pool-X-thread-Y", where X is a number going up all the time, and Y a number between 1 and 20.
• CPU usage will reach 100% and only a restart helps.

Resolution

If the problem is code creating too many thread pools, fix it to re-use the thread pools instead. If the Byteman script points to EAP code doing this, please file a support case with details.

If done with an executor and its thread pool, be sure to call shutdown() on them to so they can be cleaned up.

If the problem is the JVM bug, upgrade to JDK 1.6u21 or later.

Root Cause

• The application is creating new executors repeatedly rather than reusing them and so new executor pools with new threads are created repeatedly.
• Threads leak with async RESTEasy client calls

Diagnostic Steps

Use Byteman with the following rules, to identify any code creating an excessive number of thread pools

RULE debug DefaultThreadFactory creation
CLASS java.util.concurrent.Executors$DefaultThreadFactory
METHOD <init>
AT EXIT
IF TRUE
DO traceStack("GSS creating DefaultThreadFactory: " + $0.namePrefix)
ENDRULE

RULE debug PrivilegedThreadFactory creation
CLASS java.util.concurrent.Executors$PrivilegedThreadFactory
METHOD <init>
AT EXIT
IF TRUE
DO traceStack("GSS creating PrivilegedThreadFactory: " + $0.namePrefix)
ENDRULE

If that is no occurring, another possibility is a Content from bugs.sun.com is not included.JVM bug. Note the original report states Solaris only, but comments below indicate it also affects Linux systems. The workaround mentioned in the bug seems to work for most people but not always. Hence the recommendation is not to use the workaround, but to upgrade the JDK to at least 1.6.0_21 (preferably to the very latest 1.6.0_XX of course)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.