EAP 7 becomes slow with many weld-preloader threads

Solution Unverified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP) 7.x

Issue

  • JBoss is becoming slow and in thread dumps we see many weld-preloader threads that have no stack:
"weld-preloader-5" #4512 daemon prio=5 os_prio=0 tid=0x00007f1eec3af800 nid=0x242bd runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   Locked ownable synchronizers:
	- None

Resolution

  • Disable any java agent or set -Dorg.jboss.weld.bootstrap.preloaderThreadPoolSize=0 in your JVM options to disable the preloader threads and avoid their frequent thread creation and destruction. (org.jboss.weld.bootstrap.preloaderThreadPoolSize defaults to #ofCPUcores - 1 but weld may also create as many as 1 such temporary thread pool per jar in the application)

Root Cause

  • A java agent is in use, which is adding extra operations and synchronization in JVMTI calls during thread creation/destruction. That can be seen in native thread dumps:
Thread 1284 (Thread 0x7f99ba56f700 (LWP 18316)):
#0  0x00007f9bddf536d5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9bdd095023 in os::PlatformEvent::park() () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/server/libjvm.so
#2  0x00007f9bdcf1e30c in JvmtiRawMonitor::SimpleEnter(Thread*) () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/server/libjvm.so
#3  0x00007f9bdcf1e997 in JvmtiRawMonitor::raw_enter(Thread*) () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/server/libjvm.so
#4  0x00007f9bdcef73c5 in JvmtiEnv::RawMonitorEnter(JvmtiRawMonitor*) () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/server/libjvm.so
#5  0x00007f9bd7bc21f8 in debugMonitorEnter () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/libjdwp.so
#6  0x00007f9bd7bae4d1 in event_callback () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/libjdwp.so
#7  0x00007f9bd7baf314 in cbThreadStart () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/libjdwp.so
#8  0x00007f9bdcf0c00f in JvmtiExport::post_thread_start(JavaThread*) () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/server/libjvm.so
#9  0x00007f9bdd1ed585 in JavaThread::run() () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/server/libjvm.so
#10 0x00007f9bdd09c198 in java_start(Thread*) () from /sam/thirdparty/tools/linux/jdk1.8.0_181/jre/lib/amd64/server/libjvm.so
#11 0x00007f9bddf4fdc5 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f9bdd86473d in clone () from /lib64/libc.so.6
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.