CLI operations hang during transaction probe in JBoss EAP 6.4

Solution Verified - Updated 2 Aug 2024

Environment

JBoss Enterprise Application Platform (EAP) 6.4.4

Issue

We issue a transaction probe via CLI:
```
  /subsystem=transactions/log-store=log-store:probe()
```
And then other CLI operations are blocked for some time until the probe completes. The transaction probe is stalled like so:

"management-handler-thread - 7" prio=10 tid=0x00007f05d4118800 nid=0xbd29 waiting for monitor entry [0x00007f056fe8d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.periodicWorkFirstPass(XARecoveryModule.java:138)
	- waiting to lock <0x00000006ea3852b8> (a com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.getNewXAResource(XARecoveryModule.java:244)
	at com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.getNewXAResource(XARecoveryModule.java:275)
	at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.getNewXAResource(XAResourceRecord.java:1199)
	at com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord.restore_state(XAResourceRecord.java:963)
	at com.arjuna.ats.arjuna.coordinator.BasicAction.restore_state(BasicAction.java:1118)
	at com.arjuna.ats.arjuna.coordinator.BasicAction.activate(BasicAction.java:488)
	- locked <0x0000000787e71cc8> (a com.arjuna.ats.arjuna.AtomicAction)
	at com.arjuna.ats.arjuna.coordinator.BasicAction.activate(BasicAction.java:451)
	at com.arjuna.ats.arjuna.tools.osb.mbean.ActionBean$GenericAtomicActionWrapper.activate(ActionBean.java:374)
	at com.arjuna.ats.arjuna.tools.osb.mbean.ActionBean.createWrapper(ActionBean.java:98)
	at com.arjuna.ats.arjuna.tools.osb.mbean.ActionBean.<init>(ActionBean.java:84)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at com.arjuna.ats.arjuna.tools.osb.mbean.UidWrapper.createMBean(UidWrapper.java:191)
	at com.arjuna.ats.arjuna.tools.osb.mbean.ObjStoreBrowser.createBean(ObjStoreBrowser.java:455)
	at com.arjuna.ats.arjuna.tools.osb.mbean.ObjStoreBrowser.probe(ObjStoreBrowser.java:408)
	- locked <0x00000006eb022c78> (a com.arjuna.ats.arjuna.tools.osb.mbean.ObjStoreBrowser)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:75)
	at sun.reflect.GeneratedMethodAccessor484.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:279)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)
	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
	at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
	at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
	at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
	at org.jboss.as.jmx.PluggableMBeanServerImpl$TcclMBeanServer.invoke(PluggableMBeanServerImpl.java:1417)
	at org.jboss.as.jmx.PluggableMBeanServerImpl.invoke(PluggableMBeanServerImpl.java:692)
	at org.jboss.as.txn.subsystem.LogStoreProbeHandler.probeTransactions(LogStoreProbeHandler.java:157)
	at org.jboss.as.txn.subsystem.LogStoreProbeHandler.execute(LogStoreProbeHandler.java:188)
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702)
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537)
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338)
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314)
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144)
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331)
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:201)
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.doExecute(ModelControllerClientOperationHandler.java:208)
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler.access$300(ModelControllerClientOperationHandler.java:135)
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:157)
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1$1.run(ModelControllerClientOperationHandler.java:153)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.jboss.as.controller.AccessAuditContext.doAs(AccessAuditContext.java:94)
	at org.jboss.as.controller.remote.ModelControllerClientOperationHandler$ExecuteRequestHandler$1.execute(ModelControllerClientOperationHandler.java:153)
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$2$1.doExecute(AbstractMessageHandler.java:298)
	at org.jboss.as.protocol.mgmt.AbstractMessageHandler$AsyncTaskRunner.run(AbstractMessageHandler.java:537)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Resolution

Apply JBoss EAP 6.4 Cumulative Patch (CP) 6
Address any thing causing long db response times and inflated periodic recovery operation times. Consider setting a query-timeout on your datasources.

Root Cause

This content is not included.BZ-1270281 Transaction probe needlessly locks CLI
Content from issues.jboss.org is not included.WFLY-5498
The transaction probe needlessly holds the controller lock and so blocks other CLI operations. If the transaction probe takes long (such as due to long periodic recovery operations), then other CLI operations can be stuck waiting for some time.

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.