Failed HornetQ start up stalls JBoss start up

Environment

Red Hat JBoss Enterprise Application Platform (EAP) 6.x

Issue

• HornetQ start up does not appear to complete and JBoss start up appears to hang as a result.
• A CLI service dump shows things waiting for HornetQ to start, which is not found.
• Typically, log messages like the following would indicate a complete HornetQ start up, but these are not found:

INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 57) HQ221007: Server is now live
INFO  [org.hornetq.core.server] (ServerService Thread Pool -- 57) HQ221001: HornetQ Server version 2.3.12.Final (2.3.12, 123) [570610e1-cb34-11e3-95ac-7b7a6c9a8ab4]

• JBoss managed server start up is timing out and thread dumps are as following:

"Controller Boot Thread" prio=10 tid=0x00007fc750242000 nid=0x4a4 in Object.wait() [0x00007fc73bffe000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007f5c222f0> (a org.jboss.as.server.BootstrapImpl$ShutdownHook)
	at java.lang.Thread.join(Thread.java:1284)
	- locked <0x00000007f5c222f0> (a org.jboss.as.server.BootstrapImpl$ShutdownHook)
	at java.lang.Thread.join(Thread.java:1358)
	at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
	at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
	at java.lang.Shutdown.runHooks(Shutdown.java:123)
	at java.lang.Shutdown.sequence(Shutdown.java:167)
	at java.lang.Shutdown.exit(Shutdown.java:212)
	- locked <0x00000007f5c22628> (a java.lang.Class for java.lang.Shutdown)
	at java.lang.Runtime.exit(Runtime.java:109)
	at java.lang.System.exit(System.java:962)
	at org.jboss.as.server.SystemExiter$DefaultExiter.exit(SystemExiter.java:114)
	at org.jboss.as.server.SystemExiter.logAndExit(SystemExiter.java:96)
	at org.jboss.as.server.ServerService.boot(ServerService.java:360)
	at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:265)
	at java.lang.Thread.run(Thread.java:748)

   Locked ownable synchronizers:
	- None

"ServerService Thread Pool -- 66" prio=10 tid=0x00007fc75c08d800 nid=0x527 in Object.wait() [0x00007fc65653d000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000007e5273290> (a java.util.concurrent.atomic.AtomicBoolean)
	at java.lang.Object.wait(Object.java:503)
	at org.jboss.as.messaging.HornetQActivationService.waitForActivation(HornetQActivationService.java:139)
	- locked <0x00000007e5273290> (a java.util.concurrent.atomic.AtomicBoolean)
	at org.jboss.as.messaging.HornetQActivationService.access$000(HornetQActivationService.java:56)
	at org.jboss.as.messaging.HornetQActivationService$1.run(HornetQActivationService.java:129)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
	at java.lang.Thread.run(Thread.java:748)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

   Locked ownable synchronizers:
	- <0x00000007e47621f0> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Resolution

Address any underlying exceptions thrown during HornetQ start up.

Root Cause

An exception raised during HornetQ start up is uncaught and silently fails its start up to result in a hang.

Diagnostic Steps

A byteman rule like the following can help log any uncaught exceptions contributing to the issue:

RULE ExposeThrowable
CLASS java.lang.Throwable
METHOD <init>
AT EXIT
IF TRUE
DO
System.out.println("*** RHT expose throwable ***");
$0.printStackTrace();
ENDRULE

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.