JBoss EAP 6/7 startup/shutdown doesn't complete

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 6.x
  • 7.x

Issue

• Startup or Shutdown does not complete on JBoss EAP 6 or 7.
• Thread dumps show component start ups in the MSC service threads stalled waiting for other components to start, which can't because the MSC thread pool is exhausted:

"MSC service thread 1-1" prio=6 tid=0x000000000a189800 nid=0x1200 in Object.wait() [0x000000000a92d000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.jboss.as.ee.component.BasicComponent.waitForComponentStart(BasicComponent.java:109)

• JBoss stuck after restart. After restart of JBoss which is running in stand alone mode on one of our server we are unable to access our application running on 8080 port. start script has no port set which makes it run on 8080 default port which was running fine earlier but after restart with no change in any configuration or deployed application server is stuck in logs.

Resolution

• Increase the MSC service thread pool size through the org.jboss.server.bootstrap.maxThreads system property.
• In EAP 6.4.9+ the additional jboss.msc.max.container.threads system property is required.

Root Cause

• Threads dumps show the MSC service thread pool is exhausted with the threads waiting for more tasks to be executed through this same pool.
• The MSC service thread pool defaults to 2 threads per CPU core, which for some environments and deployments may be too small.
• EAP 6 start up stalls on EJB creation
• EAP 6 start up stalls in SecurityUtil.createMethodAndCacheIt
• EAP 6 deployment stalls and throws warning when validating beans.xml
• Spring start up spends excessive time in dependency resolution
• EAP 6 shutdown stalls when using an Executor pool for JBossWeb
• JBoss shutdown hangs when using a queueless executor
• JBoss shutdown is stalled in Connector.pause
• Shutdown or ndeployment hangs in DistributableSessionManager.stop until HTTP requests finish processing
• Shutdown stalls on XARecoveryModule.removeXAResourceRecoveryHelper
• JBoss shutdown stalls on XARecoveryModule.waitForScanState
• Failed HornetQ start up stalls JBoss start up
• JBoss shutdown hangs on IBM MQ ServerSessionImpl
• EAP 6 takes long time to boot loading a CRL file
• JBoss hangs during startup if application calls System.exit() from initialisation code
• JBoss EAP start up doesn't complete with 1024 ServerService threads in use
• EAP 7 shutdown stalls with in process requests
• EAP 7 deployment spends a lot of time in weld deployment processing
• Slowness / Memory issues when starting JBoss with many JSF 2 apps on java 8 where "ZipFileInflaterInputStream" count is growing and pending finalization
• JSF application start up hangs on EAP 7.4.15+

Diagnostic Steps

• Troubleshoot using thread dumps, garbage collection logging, and additional steps as described in Java application unresponsive

• Try capturing a service dump at the time of the issue like so:

      /host=hostname/server=servername/core-service=service-container:dump-services
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.