JBoss deadlocks between MicroRemoteClientInvoker.establishLease and Client.notifyListeners

Environment

• JBoss Enterprise Application Platform (EAP)
• JBoss SOA-P 5.3.0 and earlier

Issue

• JBoss becomes unresponsive with threads deadlocked like the following:

"Thread-1":
	at org.jboss.remoting.MicroRemoteClientInvoker.establishLease(MicroRemoteClientInvoker.java:507)
	- waiting to lock <0xfffffd7cb87677c0> (a java.lang.Object)
	at org.jboss.remoting.Client.setupClientLease(Client.java:2056)
	- locked <0xfffffd7cb8443b08> (a java.lang.Object)
	at org.jboss.remoting.Client.connect(Client.java:1918)
	at org.jboss.remoting.Client.connect(Client.java:737)
	at org.jboss.jms.client.remoting.JMSRemotingConnection$1.run(JMSRemotingConnection.java:374)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.jboss.jms.client.remoting.JMSRemotingConnection.start(JMSRemotingConnection.java:368)
	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.org$jboss$jms$client$delegate$ClientConnectionFactoryDelegate$createConnectionDelegate$aop(ClientConnectionFactoryDelegate.java:175)
	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.invokeTarget(ClientConnectionFactoryDelegate$createConnectionDelegate_N3019492359065420858.java)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:111)
	at org.jboss.jms.client.container.StateCreationAspect.handleCreateConnectionDelegate(StateCreationAspect.java:80)
	at org.jboss.aop.advice.org.jboss.jms.client.container.StateCreationAspect_z_handleCreateConnectionDelegate_132715097.invoke(StateCreationAspect_z_handleCreateConnectionDelegate_132715097.java)
	at org.jboss.aop.joinpoint.MethodInvocation.invokeNext(MethodInvocation.java:102)
	at org.jboss.jms.client.delegate.ClientConnectionFactoryDelegate.createConnectionDelegate(ClientConnectionFactoryDelegate.java)
	at org.jboss.jms.client.JBossConnectionFactory.createConnectionInternal(JBossConnectionFactory.java:205)
	at org.jboss.jms.client.JBossConnectionFactory.createTopicConnection(JBossConnectionFactory.java:115)
        ...
	at java.lang.Thread.run(Thread.java:662)
"Timer-1":
	at org.jboss.remoting.Client.notifyListeners(Client.java:1873)
	- waiting to lock <0xfffffd7cb8443b08> (a java.lang.Object)
	at org.jboss.remoting.LeasePinger.stopPing(LeasePinger.java:134)
	at org.jboss.remoting.MicroRemoteClientInvoker.terminateLease(MicroRemoteClientInvoker.java:434)
	- locked <0xfffffd7cb87677c0> (a java.lang.Object)
	at org.jboss.remoting.ConnectionValidator$WaitOnConnectionCheckTimerTask.run(ConnectionValidator.java:1081)
	at java.util.TimerThread.mainLoop(Timer.java:512)
	at java.util.TimerThread.run(Timer.java:462)

Resolution

• Upgrade to EAP 5.2.0
• Upgrade to SOA-P 5.3.1 (based on EAP 5.2.0)
• Acquire a one-off for your version if available:
  • This content is not included.5.1.2
  • This content is not included.5.1.1
  • This content is not included.4.3_CP10

Root Cause

• Content from issues.jboss.org is not included.JBREM-1297

Diagnostic Steps

• Troubleshoot using thread dumps and additional steps mentioned in Java application unresponsive

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.