JBoss becomes unresponsive with many running threads that have no stack

Solution Unverified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)

Issue

  • JBoss becomes unresponsive. Thread dumps taken at the time show many threads (20000 or more) that have no thread stack, for example:
"Thread-2597636" daemon prio=10 tid=0x00007f1f965d7000 nid=0x76b0 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

Resolution

  • Disable any java agents to avoid the extra bottlenecks introduced at thread start and end

Root Cause

  • The stackless threads are stalled in JVM level code during their start or end
  • A java agent is used that is enabling JVMTI reporting for thread start and end events at least. Underneath the JVMTI API calls are calls to check for various things that might need doing in response to a thread start or end, most notably negotiating with a debugger. These calls are guarded with a monitor to ensure exclusive access. The running thread which holds the lock will be doing one of the many possible housekeeping events necessary at thread start. This represents a bottleneck on thread start and end whose rate of forward progress is determined by how much work the JVMTI callback has to do. Even with no events to actually respond to, the basic checks could be enough to cause a problem given a fast enough rate of thread creation.
  • Some JBoss threading configurations may frequently create new threads or certain things like ldap connections or database drivers will frequently churn up a new thread to track a timeout for each query or connection that's started. So high thread creation/destruction from such scenarios coupled with the extra bottleneck from a java agent could easily bog down the server.
  • EAP 7 becomes slow with many weld-preloader threads
  • Performance of Oracle datasource validation degrades in EAP 7.4.2+

Diagnostic Steps

  • Capture a java thread dump and check for an abundance of stackless threads
  • Capture a heap dump to clarify what those threads may be created for. Check the target of each thread to see what they should run.
  • Capture native level thread dumps via jstack -m or pstack to see where threads may be stalled at a native layer. For instance, threads stalled in thread start and end can look like so:
Thread 2734 (Thread 0x7f6a741c4700 (LWP 51097)):
#0  0x0000003c0c20b63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f6f956d475b in os::PlatformEvent::park() () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#2  0x00007f6f955c436c in JvmtiRawMonitor::raw_enter(Thread*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#3  0x00007f6f955a08f2 in JvmtiEnv::RawMonitorEnter(JvmtiRawMonitor*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#4  0x00007f6f947a1338 in debugMonitorEnter () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#5  0x00007f6f9479a325 in threadControl_onEventHandlerExit () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#6  0x00007f6f9478e001 in event_callback () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#7  0x00007f6f9478ed00 in cbThreadEnd () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#8  0x00007f6f955b6a72 in JvmtiExport::post_thread_end(JavaThread*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#9  0x00007f6f9580b848 in JavaThread::exit () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#10 0x00007f6f9580bcec in JavaThread::thread_main_inner() () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#11 0x00007f6f9580c030 in JavaThread::run() () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#12 0x00007f6f956dbd02 in java_start(Thread*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#13 0x0000003c0c207a51 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003c0bee893d in clone () from /lib64/libc.so.6

Thread 2755 (Thread 0x7f6a8af30700 (LWP 48004)):
#0  0x0000003c0c20b63c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f6f956d475b in os::PlatformEvent::park() () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#2  0x00007f6f955c436c in JvmtiRawMonitor::raw_enter(Thread*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#3  0x00007f6f955a08f2 in JvmtiEnv::RawMonitorEnter(JvmtiRawMonitor*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#4  0x00007f6f947a1338 in debugMonitorEnter () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#5  0x00007f6f9478debb in event_callback () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#6  0x00007f6f9478ef00 in cbThreadStart () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#7  0x00007f6f955b6802 in JvmtiExport::post_thread_start(JavaThread*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#8  0x00007f6f9580c0a8 in JavaThread::run() () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#9  0x00007f6f956dbd02 in java_start(Thread*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#10 0x0000003c0c207a51 in start_thread () from /lib64/libpthread.so.0
#11 0x0000003c0bee893d in clone () from /lib64/libc.so.6

Thread 1073 (Thread 0x7f6a1ef77700 (LWP 57607)):
#0  0x00007f6f955133d3 in jni_IsSameObject () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#1  0x00007f6f9478705e in classTrack_processUnloads () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#2  0x00007f6f9478e0a3 in event_callback () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#3  0x00007f6f9478ef00 in cbThreadStart () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/libjdwp.so
#4  0x00007f6f955b6802 in JvmtiExport::post_thread_start(JavaThread*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#5  0x00007f6f9580c0a8 in JavaThread::run() () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#6  0x00007f6f956dbd02 in java_start(Thread*) () from /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.91.x86_64/jre/lib/amd64/server/libjvm.so
#7  0x0000003c0c207a51 in start_thread () from /lib64/libpthread.so.0
#8  0x0000003c0bee893d in clone () from /lib64/libc.so.6
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.