Shutdown stalls on XARecoveryModule.removeXAResourceRecoveryHelper

Environment

• JBoss Enterprise Application Platform (EAP) 5.x

Issue

• JBoss shutdown is stalled in the following:'

"JBoss Shutdown Hook" daemon prio=6 tid=0x000000000c507000 nid=0x143c waiting for monitor entry [0x000000001621d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.removeXAResourceRecoveryHelper(XARecoveryModule.java:93)
	- waiting to lock <0x00000000c6feb518> (a java.util.LinkedList)
	at com.arjuna.ats.jbossatx.jta.TransactionManagerService.removeXAResourceRecovery(TransactionManagerService.java:718)
	at org.jboss.resource.connectionmanager.ManagedConnectionFactoryDeployment.stopService(ManagedConnectionFactoryDeployment.java:526)

• The lock is held by the periodic recovery thread:

"Thread-20" prio=6 tid=0x0000000011e90000 nid=0x1cb0 runnable [0x0000000015cde000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at com.microsoft.sqlserver.jdbc.TDSChannel.read(IOBuffer.java:1647)
	at com.microsoft.sqlserver.jdbc.TDSReader.readPacket(IOBuffer.java:3694)
	- locked <0x00000000c9596778> (a com.microsoft.sqlserver.jdbc.TDSReader)
	at com.microsoft.sqlserver.jdbc.TDSCommand.startResponse(IOBuffer.java:5022)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.doExecutePreparedStatement(SQLServerPreparedStatement.java:388)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement$PrepStmtExecCmd.doExecute(SQLServerPreparedStatement.java:340)
	at com.microsoft.sqlserver.jdbc.TDSCommand.execute(IOBuffer.java:4575)
	at com.microsoft.sqlserver.jdbc.SQLServerConnection.executeCommand(SQLServerConnection.java:1400)
	- locked <0x00000000ca356f08> (a java.lang.Object)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeCommand(SQLServerStatement.java:179)
	at com.microsoft.sqlserver.jdbc.SQLServerStatement.executeStatement(SQLServerStatement.java:154)
	at com.microsoft.sqlserver.jdbc.SQLServerPreparedStatement.execute(SQLServerPreparedStatement.java:322)
	at com.microsoft.sqlserver.jdbc.SQLServerXAResource.DTC_XA_Interface(SQLServerXAResource.java:534)
	at com.microsoft.sqlserver.jdbc.SQLServerXAResource.recover(SQLServerXAResource.java:723)
	at org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.recover(XAManagedConnection.java:294)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.xaRecovery(XARecoveryModule.java:787)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.resourceInitiatedRecoveryForRecoveryHelpers(XARecoveryModule.java:726)
	- locked <0x00000000c6feb518> (a java.util.LinkedList)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.bottomUpRecovery(XARecoveryModule.java:616)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkSecondPass(XARecoveryModule.java:195)
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.doWorkInternal(PeriodicRecovery.java:799)
	at com.arjuna.ats.internal.arjuna.recovery.PeriodicRecovery.run(PeriodicRecovery.java:412)

Resolution

• Set a query-timeout on your datasources to ensure proper timeout of any long running db operations:

    <query-timeout>#ofseconds</query-timeout>
  

Root Cause

• A possible networking issue or long running db operation is delaying the PeriodicRecovery thread and in turn stalling shutdown. No timeout is used by default.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.