JBoss Messaging stalls startup synchronizing on an aop AspectManager

Environment

• JBoss Enterprise Application Platform (EAP) 5

Issue

• Our JBoss start never completes and is perpetually stalled. Thread dumps show the main thread hanging in ServerConnectionFactoryEndpoint.createConnectionDelegateInternal() along with others, for example:

"main" prio=10 tid=0x00007f2e340e1800 nid=0x5199 waiting for monitor entry [0x00007f2e30bf6000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jboss.aop.AspectManager.getAdvisor(AspectManager.java:728)
	- waiting to lock <0x00000000a6dc7f78> (a org.jboss.aop.AspectManager)
	at org.jboss.jms.client.delegate.ClientConnectionDelegate.<clinit>(ClientConnectionDelegate.java)
	at org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegateInternal(ServerConnectionFactoryEndpoint.java:296)
	- locked <0x00000000a52dc1f8> (a org.jboss.aop.asintegration.jboss5.ScopedVFSClassLoaderDomain)
	at org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegate(ServerConnectionFactoryEndpoint.java:170
...

"WorkManager(2)-1" daemon prio=10 tid=0x00007f2dc90db800 nid=0x52a4 in Object.wait() [0x00007f2e08b56000]
   java.lang.Thread.State: RUNNABLE
	at org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegateInternal(ServerConnectionFactoryEndpoint.java:296)
	- locked <0x00000000a6dc7f78> (a org.jboss.aop.AspectManager)
	at org.jboss.jms.server.endpoint.ServerConnectionFactoryEndpoint.createConnectionDelegate(ServerConnectionFactoryEndpoint.java:170)
...

Resolution

• This AspectManager synchronization causing the issue here is no longer needed at this point as it was implemented to avoid an AOP level deadlock that is now resolved within AOP.
• The following one-offs are available to fix such issues by removing the old AspectManager locking done here:
  • This content is not included.5.1.1
  • This content is not included.5.1.2
  • This content is not included.5.2.0

Root Cause

• The two deadlocked threads are both trying to initialize ClientConnectionDelegate, which requires synchronization at the native level. The main thread grabs the AspectManager lock after it entered the ClientConnectionDelegate.<clinit> calls and so it can deadlock with any other thread making similar calls with inverse lock ordering.
• Similar synchronizing on an AspectManager is also implemented in these classes and could cause issues:
  • org.jboss.jms.server.endpointServerConnectionFactoryEndpoint.java
  • org.jboss.jms.server.endpoint.ServerSessionEndpoint.java
  • org.jboss.jms.server.endpoint.ServerConnectionEndpoint.java
  • org.jboss.jms.server.connectionfactory.ConnectionFactoryJNDIMapper.java
  • org.jboss.jms.server.connectionmanager.SimpleConnectionManager.java
• Content from issues.jboss.org is not included.JBMESSAGING-1941

Diagnostic Steps

• Troubleshoot with thread dumps and additional steps mentioned in JBoss is slow to start
• Capture pstack from JBoss along with thread dumps from this time to check that the thread is indeed stuck in native level waits from initialization calls:

Thread 2 (Thread 0x7f2e08b58700 (LWP 21156)):
#0  0x0000003b3e40b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f2e3bd4a47e in os::PlatformEvent::park() () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
#2  0x00007f2e3bd3b4fa in ObjectMonitor::wait(long, bool, Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
#3  0x00007f2e3be215f0 in ObjectSynchronizer::waitUninterruptibly(Handle, long, Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
#4  0x00007f2e3bae715b in instanceKlass::initialize_impl(instanceKlassHandle, Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
#5  0x00007f2e3bae677a in instanceKlass::initialize(Thread*) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so
#6  0x00007f2e3bb0dc5c in InterpreterRuntime::_new(JavaThread*, constantPoolOopDesc*, int) () from /opt/app/jvm/jdk1.6.0_38/jre/lib/amd64/server/libjvm.so

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.