Deadlock during EJB client discovery leads to high thread stalls on org.jboss.remoting3.ConnectionInfo.getConnection

Environment

• JBoss Enterprise Application Platform
  • 7.4.x
  • 8.0.x

Issue

• We see JBoss become unresponsive and thread dumps show a deadlock like below between org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt.onCancel and org.jboss.remoting3.EndpointImpl.connect:

"XNIO-1 task-3":
  waiting to lock monitor 0x000055b2ff1b7480 (object 0x0000000087f7a958, a java.util.Collections$SynchronizedRandomAccessList),
  which is held by "default task-1"
"default task-1":
  waiting to lock monitor 0x000055b2eb1dfc80 (object 0x0000000084cf7ea8, a java.lang.Object),
  which is held by "XNIO-1 task-3"

Java stack information for the threads listed above:
===================================================
"XNIO-1 task-3":
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt.onCancel(RemotingEJBDiscoveryProvider.java:613)
	- waiting to lock <0x0000000087f7a958> (a java.util.Collections$SynchronizedRandomAccessList)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt$1.handleDone(RemotingEJBDiscoveryProvider.java:397)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt$1.handleDone(RemotingEJBDiscoveryProvider.java:380)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:856)
	at org.xnio.IoUtils$2.execute(IoUtils.java:71)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:829)
	at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:402)
	at org.xnio.AbstractIoFuture$NotifierState.doNotifyDone(AbstractIoFuture.java:383)
	at org.xnio.AbstractIoFuture$NestedState.notifyDone(AbstractIoFuture.java:95)
	at org.xnio.AbstractIoFuture.setResult(AbstractIoFuture.java:755)
	at org.xnio.FutureResult.setResult(FutureResult.java:75)
	at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:522)
	at org.jboss.remoting3.EndpointImpl$3.handleDone(EndpointImpl.java:506)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:856)
	at org.xnio.IoUtils$2.execute(IoUtils.java:71)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:829)
	at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:402)
	at org.xnio.AbstractIoFuture$NotifierState.doNotifyDone(AbstractIoFuture.java:383)
	at org.xnio.AbstractIoFuture$NestedState.notifyDone(AbstractIoFuture.java:95)
	at org.xnio.AbstractIoFuture.setResult(AbstractIoFuture.java:755)
	at org.xnio.FutureResult.setResult(FutureResult.java:75)
	at org.jboss.remoting3.ConnectionInfo$1.handleDone(ConnectionInfo.java:326)
	at org.jboss.remoting3.ConnectionInfo$1.handleDone(ConnectionInfo.java:316)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:856)
	at org.xnio.IoUtils$2.execute(IoUtils.java:71)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:829)
	at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:402)
	at org.xnio.AbstractIoFuture$NotifierState.doNotifyDone(AbstractIoFuture.java:383)
	at org.xnio.AbstractIoFuture$NestedState.notifyDone(AbstractIoFuture.java:100)
	at org.xnio.AbstractIoFuture.setResult(AbstractIoFuture.java:755)
	at org.xnio.FutureResult.setResult(FutureResult.java:75)
	at org.jboss.remoting3.EndpointImpl$4.setResult(EndpointImpl.java:641)
	- locked <0x0000000084cf7ea8> (a java.lang.Object)
	at org.jboss.remoting3.EndpointImpl$4.setResult(EndpointImpl.java:602)
	at org.xnio.IoUtils$ResultNotifier.handleDone(IoUtils.java:725)
	at org.xnio.IoUtils$ResultNotifier.handleDone(IoUtils.java:714)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:856)
	at org.xnio.IoUtils$2.execute(IoUtils.java:71)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:829)
	at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:402)
	at org.xnio.AbstractIoFuture$NotifierState.doNotifyDone(AbstractIoFuture.java:383)
	at org.xnio.AbstractIoFuture$NestedState.notifyDone(AbstractIoFuture.java:100)
	at org.xnio.AbstractIoFuture.setResult(AbstractIoFuture.java:755)
	at org.xnio.FutureResult.setResult(FutureResult.java:75)
	at org.jboss.remoting3.remote.ClientConnectionOpenListener$Authentication.lambda$handleEvent$2(ClientConnectionOpenListener.java:747)
	at org.jboss.remoting3.remote.ClientConnectionOpenListener$Authentication$$Lambda$275/0x0000000840605840.run(Unknown Source)
	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:993)
	at org.jboss.remoting3.EndpointImpl$TrackingExecutor$$Lambda$272/0x0000000840606c40.run(Unknown Source)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.lang.Thread.run(java.base@11.0.23/Thread.java:829)
"default task-1":
	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:586)
	- waiting to lock <0x0000000084cf7ea8> (a java.lang.Object)
	at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:565)
	at org.jboss.remoting3.ConnectionInfo$None.getConnection(ConnectionInfo.java:82)
	- locked <0x0000000087f791a0> (a org.jboss.remoting3.ConnectionInfo)
	at org.jboss.remoting3.ConnectionInfo.getConnection(ConnectionInfo.java:55)
	at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:499)
	at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:445)
	at org.jboss.remoting3.UncloseableEndpoint.getConnectedIdentity(UncloseableEndpoint.java:52)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider.getConnectedIdentityUsingClusterEffective(RemotingEJBDiscoveryProvider.java:342)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt.connectAndDiscover(RemotingEJBDiscoveryProvider.java:436)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt.countDown(RemotingEJBDiscoveryProvider.java:548)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt$1.handleCancelled(RemotingEJBDiscoveryProvider.java:382)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt$1.handleCancelled(RemotingEJBDiscoveryProvider.java:380)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:204)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:856)
	at org.xnio.IoUtils$2.execute(IoUtils.java:71)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:829)
	at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:402)
	at org.xnio.AbstractIoFuture$NotifierState.doNotifyCancelled(AbstractIoFuture.java:393)
	at org.xnio.AbstractIoFuture$NestedState.notifyCancelled(AbstractIoFuture.java:150)
	at org.xnio.AbstractIoFuture.setCancelled(AbstractIoFuture.java:779)
	at org.xnio.FutureResult.setCancelled(FutureResult.java:94)
	at org.jboss.remoting3.EndpointImpl$3.handleCancelled(EndpointImpl.java:508)
	at org.jboss.remoting3.EndpointImpl$3.handleCancelled(EndpointImpl.java:506)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:204)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:856)
	at org.xnio.IoUtils$2.execute(IoUtils.java:71)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:829)
	at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:402)
	at org.xnio.AbstractIoFuture$NotifierState.doNotifyCancelled(AbstractIoFuture.java:393)
	at org.xnio.AbstractIoFuture$NestedState.notifyCancelled(AbstractIoFuture.java:150)
	at org.xnio.AbstractIoFuture.setCancelled(AbstractIoFuture.java:779)
	at org.xnio.FutureResult.setCancelled(FutureResult.java:94)
	at org.jboss.remoting3.ConnectionInfo$1.handleCancelled(ConnectionInfo.java:318)
	at org.jboss.remoting3.ConnectionInfo$1.handleCancelled(ConnectionInfo.java:316)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:204)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:856)
	at org.xnio.IoUtils$2.execute(IoUtils.java:71)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:829)
	at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:402)
	at org.xnio.AbstractIoFuture$NotifierState.doNotifyCancelled(AbstractIoFuture.java:393)
	at org.xnio.AbstractIoFuture$NestedState.notifyCancelled(AbstractIoFuture.java:150)
	at org.xnio.AbstractIoFuture.setCancelled(AbstractIoFuture.java:779)
	at org.xnio.FutureResult.setCancelled(FutureResult.java:94)
	at org.jboss.remoting3.EndpointImpl$4.setCancelled(EndpointImpl.java:610)
	at org.xnio.IoUtils$ResultNotifier.handleCancelled(IoUtils.java:717)
	at org.xnio.IoUtils$ResultNotifier.handleCancelled(IoUtils.java:714)
	at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:204)
	at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:856)
	at org.xnio.IoUtils$2.execute(IoUtils.java:71)
	at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:829)
	at org.xnio.AbstractIoFuture$NotifierState.doNotify(AbstractIoFuture.java:402)
	at org.xnio.AbstractIoFuture$NotifierState.doNotifyCancelled(AbstractIoFuture.java:393)
	at org.xnio.AbstractIoFuture$NestedState.notifyCancelled(AbstractIoFuture.java:150)
	at org.xnio.AbstractIoFuture.setCancelled(AbstractIoFuture.java:779)
	at org.xnio.FutureResult.setCancelled(FutureResult.java:94)
	at org.jboss.remoting3.remote.RemoteConnectionProvider$2.cancel(RemoteConnectionProvider.java:173)
	at org.xnio.AbstractIoFuture$CancellableState.doCancel(AbstractIoFuture.java:474)
	at org.xnio.AbstractIoFuture$NestedState.cancel(AbstractIoFuture.java:174)
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:796)
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:35)
	at org.xnio.AbstractIoFuture$CancellableState.doCancel(AbstractIoFuture.java:474)
	at org.xnio.AbstractIoFuture$NestedState.cancel(AbstractIoFuture.java:174)
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:796)
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:35)
	at org.xnio.AbstractIoFuture$CancellableState.doCancel(AbstractIoFuture.java:474)
	at org.xnio.AbstractIoFuture$NestedState.cancel(AbstractIoFuture.java:174)
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:796)
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:35)
	at org.xnio.AbstractIoFuture$CancellableState.doCancel(AbstractIoFuture.java:474)
	at org.xnio.AbstractIoFuture$NestedState.cancel(AbstractIoFuture.java:174)
	at org.xnio.AbstractIoFuture.cancel(AbstractIoFuture.java:796)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt$$Lambda$2030/0x00000008425a4c40.run(Unknown Source)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt.cancel(RemotingEJBDiscoveryProvider.java:605)
	- locked <0x0000000087f7a958> (a java.util.Collections$SynchronizedRandomAccessList)
	at org.wildfly.discovery.impl.AggregateDiscoveryProvider$AggregatingDiscoveryRequest.cancel(AggregateDiscoveryProvider.java:89)
	at org.wildfly.discovery.Discovery$BlockingQueueServicesQueue.close(Discovery.java:296)
	at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.doAnyDiscovery(DiscoveryEJBClientInterceptor.java:553)
	at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.executeDiscovery(DiscoveryEJBClientInterceptor.java:426)
	at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.handleInvocation(DiscoveryEJBClientInterceptor.java:110)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
	at org.jboss.ejb.client.NamingEJBClientInterceptor.handleInvocation(NamingEJBClientInterceptor.java:69)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)
	at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.lambda$handleInvocation$0(AuthenticationContextEJBClientInterceptor.java:45)
	at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor$$Lambda$2026/0x0000000842466840.get(Unknown Source)
	at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.call(AuthenticationContextEJBClientInterceptor.java:59)
	at org.jboss.ejb.client.AuthenticationContextEJBClientInterceptor.handleInvocation(AuthenticationContextEJBClientInterceptor.java:44)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:514)

• Other stalled threads build up in org.jboss.remoting3.ConnectionInfo.getConnection as a result:

"default task-166" #2108 prio=5 os_prio=0 cpu=10.01ms elapsed=159.94s tid=0x000055b2fc25c000 nid=0x151be waiting for monitor entry  [0x00007f2cafd6d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jboss.remoting3.ConnectionInfo.getConnection(ConnectionInfo.java:52)
	- waiting to lock <0x000000008bdcd558> (a org.jboss.remoting3.ConnectionInfo)
	at org.jboss.remoting3.EndpointImpl.doGetConnection(EndpointImpl.java:499)
	at org.jboss.remoting3.EndpointImpl.getConnectedIdentity(EndpointImpl.java:445)
	at org.jboss.remoting3.UncloseableEndpoint.getConnectedIdentity(UncloseableEndpoint.java:52)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider.getConnectedIdentityUsingClusterEffective(RemotingEJBDiscoveryProvider.java:342)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider$DiscoveryAttempt.connectAndDiscover(RemotingEJBDiscoveryProvider.java:436)
	at org.jboss.ejb.protocol.remote.RemotingEJBDiscoveryProvider.discover(RemotingEJBDiscoveryProvider.java:207)
	at org.jboss.ejb.protocol.remote.RemoteEJBDiscoveryConfigurator.lambda$configure$0(RemoteEJBDiscoveryConfigurator.java:42)
	at org.jboss.ejb.protocol.remote.RemoteEJBDiscoveryConfigurator$$Lambda$1023/0x0000000841116840.discover(Unknown Source)
	at org.wildfly.discovery.impl.AggregateDiscoveryProvider.discover(AggregateDiscoveryProvider.java:58)
	at org.wildfly.discovery.Discovery.discover(Discovery.java:122)
	at org.wildfly.discovery.Discovery.discover(Discovery.java:93)
	at org.jboss.ejb.client.DiscoveryEJBClientInterceptor.discover(DiscoveryEJBClientInterceptor.java:346)

Resolution

• Update to EAP 7.4 CP19 or later or EAP 8.0 Update 5 or later

Root Cause

• This content is not included.EJBCLIENT-536
• This content is not included.JBEAP-27521
• This content is not included.JBEAP-28241

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.