JCA dead-lock in JBoss EAP

Environment

• JBoss Enterprise Application Platform (EAP)
  • 4.3.0_CP09 and earlier
  • 5.1.0 and earlier

Issue

The following dead-lock was discovered in the jstack dumps of our application:

Found one Java-level deadlock:
=============================
"WorkManager(2)-28127396":
  waiting to lock monitor 0x0000000105b7e2a8 (object 0xfffffffb7bd52be8, a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction),
  which is held by "Thread-3"
"Thread-3":
  waiting for ownable synchronizer 0xfffffffb7bd51b58, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "WorkManager(2)-28127396"

Java stack information for the threads listed above:
===================================================
"WorkManager(2)-28127396":
        at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1879)
        - waiting to lock <0xfffffffb7bd52be8> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:105)
        at com.arjuna.ats.arjuna.AtomicAction.abort(AtomicAction.java:201)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple.commitAndDisassociate(TransactionImple.java:1395)
        at com.arjuna.ats.internal.jta.transaction.arjunacore.BaseTransaction.commit(BaseTransaction.java:134)
        at com.arjuna.ats.jbossatx.BaseTransactionManagerDelegate.commit(BaseTransactionManagerDelegate.java:87)
        at org.jboss.soa.esb.listeners.jca.EndpointProxy.endTransaction(EndpointProxy.java:384)
        at org.jboss.soa.esb.listeners.jca.EndpointProxy.delivery(EndpointProxy.java:264)
        at org.jboss.soa.esb.listeners.jca.EndpointProxy.invoke(EndpointProxy.java:145)
        at $Proxy70.onMessage(Unknown Source)
        at org.jboss.resource.adapter.jms.inflow.JmsServerSession.onMessage(JmsServerSession.java:179)
        at org.jboss.jms.client.container.ClientConsumer.callOnMessageStatic(ClientConsumer.java:166)
        at org.jboss.jms.client.container.SessionAspect.handleRun(SessionAspect.java:831)
        at org.jboss.aop.advice.org.jboss.jms.client.container.SessionAspect14.invoke(SessionAspect14.java)
        at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
        at org.jboss.jms.client.container.ClosedInterceptor.invoke(ClosedInterceptor.java:172)
        at org.jboss.aop.advice.PerInstanceInterceptor.invoke(PerInstanceInterceptor.java:105)
        at org.jboss.jms.client.delegate.ClientSessionDelegate$run_N8003352271541955702.invokeNext(ClientSessionDelegate$run_N8003352271541955702.java)
        at org.jboss.jms.client.delegate.ClientSessionDelegate.run(ClientSessionDelegate.java)
        at org.jboss.jms.client.JBossSession.run(JBossSession.java:199)
        at org.jboss.resource.adapter.jms.inflow.JmsServerSession.run(JmsServerSession.java:239)
        at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:213)
        at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:275)
        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:756)
        at java.lang.Thread.run(Thread.java:662)

"Thread-3":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xfffffffb7bd51b58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.lock(BaseWrapperManagedConnection.java:255)
        at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.end(XAManagedConnection.java:153)
        at org.jboss.resource.connectionmanager.xa.JcaXAResourceWrapper.end(JcaXAResourceWrapper.java:58)
        at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.topLevelAbort(XAResourceRecord.java:397)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3231)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.doAbort(BasicAction.java:3209)
        at com.arjuna.ats.arjuna.coordinator.BasicAction.Abort(BasicAction.java:1931)
        - locked <0xfffffffb7bd52be8> (a com.arjuna.ats.internal.jta.transaction.arjunacore.AtomicAction)
        at com.arjuna.ats.arjuna.coordinator.TwoPhaseCoordinator.cancel(TwoPhaseCoordinator.java:105)
        at com.arjuna.ats.arjuna.AtomicAction.cancel(AtomicAction.java:230)
        at com.arjuna.ats.arjuna.coordinator.TransactionReaper.check(TransactionReaper.java:201)
        at com.arjuna.ats.internal.arjuna.coordinator.ReaperThread.run(ReaperThread.java:107)

Found 1 deadlock.

Resolution

• This issue is related to the following JIRAs and fixed in EAP 4.3 CP10 / EAP 5.1.2 or later:
  • Content from issues.jboss.org is not included.[JBPAPP-5596] Incorrect handling of multi-threaded access to JCA connections
  • Content from issues.jboss.org is not included.[JBPAPP-6684] Deadlock in JCA when TM tries to commit/rollback
• Upgrade to the latest release of JBoss EAP 4.3/5.x (= EAP 4.3 CP10 / EAP 5.2.0), which do not suffer from this problem.

Diagnostic Steps

• Troubleshoot using thread dumps and additional steps mentioned in Java application unresponsive

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.