EAP 6 EJB client deadlock

Solution Unverified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 6.x
    • 7.0.0

Issue

  • Occassionally, we see a deadlock like the following in EJB client code between org.jboss.ejb.client.ClusterContext.receiverContextClosed and org.jboss.ejb.client.EJBClientContext.getNodeEJBReceiver :
"Remoting "config-based-ejb-client-endpoint" read-1":
	at java.util.Collections$SynchronizedCollection.remove(Collections.java:1580)
	- waiting to lock <0x000000078609c840> (a java.util.Collections$SynchronizedSet)
	at org.jboss.ejb.client.ClusterContext.receiverContextClosed(ClusterContext.java:414)
	at org.jboss.ejb.client.EJBClientContext.unregisterEJBReceiver(EJBClientContext.java:443)
	- locked <0x000000078608c238> (a java.util.IdentityHashMap)
	at org.jboss.ejb.client.EJBReceiverContext.close(EJBReceiverContext.java:59)
	at org.jboss.ejb.client.remoting.ChannelAssociation.notifyBrokenChannel(ChannelAssociation.java:407)
	at org.jboss.ejb.client.remoting.ChannelAssociation.access$100(ChannelAssociation.java:59)
	at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:118)
	at org.jboss.ejb.client.remoting.ChannelAssociation$1.handleClose(ChannelAssociation.java:110)
	at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:54)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:514)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:419)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:290)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:531)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:372)
	at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:437)
	at org.jboss.remoting3.remote.RemoteConnectionHandler.receiveCloseRequest(RemoteConnectionHandler.java:217)
	at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:113)
	at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:46)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
	at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
	at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
	at org.xnio.nio.NioHandle.run(NioHandle.java:90)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)

"Client09":
	at org.jboss.ejb.client.EJBClientContext.getNodeEJBReceiver(EJBClientContext.java:860)
	- waiting to lock <0x000000078608c238> (a java.util.IdentityHashMap)
	at org.jboss.ejb.client.EJBClientContext.getNodeEJBReceiver(EJBClientContext.java:846)
	at org.jboss.ejb.client.EJBClientContext.getNodeEJBReceiverContext(EJBClientContext.java:888)
	at org.jboss.ejb.client.ClusterContext.isNodeConnectedAndDeployed(ClusterContext.java:251)
	at org.jboss.ejb.client.ClusterContext.getConnectedAndDeployedNodes(ClusterContext.java:266)
	- locked <0x000000078609c840> (a java.util.Collections$SynchronizedSet)
	at org.jboss.ejb.client.ClusterContext.getEJBReceiverContext(ClusterContext.java:125)
	at org.jboss.ejb.client.ClusterContext.getEJBReceiverContext(ClusterContext.java:96)
	at org.jboss.ejb.client.EJBClientContext.getClusterEJBReceiverContext(EJBClientContext.java:958)
	at org.jboss.ejb.client.EJBClientContext.getClusterEJBReceiverContext(EJBClientContext.java:938)
	at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:108)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
	at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
	at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)

Resolution

  • Upgrade the EJB client library to 7.0.1+or 6.4.8+. The latest is recommended.

Root Cause

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.