JBoss EAP start up doesn't complete with 1024 ServerService threads in use

Environment

• JBoss Enterprise Application Platform (EAP)
  • 7.2.x
  • 7.3.x

Issue

• Our JBoss start up is timing out and failing to complete at random. Thread dumps or heap dumps in that time show 1024 ServerService threads present and in use and all stuck in wait states:

"ServerService Thread Pool -- 252" #853 prio=5 os_prio=0 tid=0x000055d2e6f95000 nid=0x13c8 in Object.wait() [0x00007f56e10f9000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at org.jboss.msc.service.StabilityMonitor.awaitStability(StabilityMonitor.java:313)
	- locked <0x0000000739d956d0> (a java.lang.Object)
	at org.jboss.msc.service.StabilityMonitor.awaitStability(StabilityMonitor.java:228)
	at org.jboss.as.weld.WeldStartCompletionService$1.run(WeldStartCompletionService.java:85)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1985)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1487)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1364)
	at java.lang.Thread.run(Thread.java:748)
	at org.jboss.threads.JBossThread.run(JBossThread.java:485)

Resolution

• Add -Djboss.threads.eqe.disable=true to your JVM options. This will switch JBoss to using a prior thread pool implementation for the ServerService Thread Pool that does not have a max.

Root Cause

• High deployment counts result in a large amount of parallel weld start up tasks that exhaust the thread pool and can't complete until more threads are available to process additional parallel tasks.
• This thread pool was previously unbounded and then This content is not included.WFCORE-3397 changed this to a Content from github.com is not included.bounded size with a fixed max of 1024 threads

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.