Many EJB client threads created during clustered remote EJB calls leads to OutOfMemoryError

Environment

• Websphere EJB client
• JBoss Enterprise Application Platform (EAP) backend
  • 6.3.1
• Scoped EJB client context

Issue

• Remote calls to stateless session beans deployed in EJB cluster results in many EJB client threads created, causing OutOfMemoryError.
• Thousands of ejb-client-cluster-node-connection-creation threads.
• Thread dump analysis shows many threads waiting on a org.jboss.ejb.client.remoting.ConnectionPool monitor. For example:

ejb-client-cluster-node-connection-creation-3-thread-1234

Monitor:
Waiting for Monitor Lock on org/jboss/ejb/client/remoting/ConnectionPool@0x0000000012A34B56

Java Stack:
at org/jboss/ejb/client/remoting/RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51(Compiled Code)) 
at org/jboss/ejb/client/remoting/RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:416(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:390(Compiled Code)) 
at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314(Compiled Code)) 
at java/util/concurrent/FutureTask.run(FutureTask.java:149(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931(Compiled Code)) 
at java/lang/Thread.run(Thread.java:773(Compiled Code)) 

• Thread dump analysis shows the ejb-client-cluster-node-connection-creation that owns the org.jboss.ejb.client.remoting.ConnectionPool monitor is attempting a connection:

Thread Name:
ejb-client-cluster-node-connection-creation-3-thread-6789

State:
Waiting on condition

Monitor:
Owns Monitor Lock on org/jboss/ejb/client/remoting/ConnectionPool@0x0000000012A34B56

Java Stack:
at sun/nio/ch/SelectorImpl.register(SelectorImpl.java:133(Compiled Code)) 
at java/nio/channels/spi/AbstractSelectableChannel.register(AbstractSelectableChannel.java:191(Compiled Code)) 
at java/nio/channels/SelectableChannel.register(SelectableChannel.java:265(Compiled Code)) 
at org/xnio/nio/WorkerThread.addChannel(WorkerThread.java:288(Compiled Code)) 
at org/xnio/nio/AbstractNioStreamChannel.start(AbstractNioStreamChannel.java:82(Compiled Code)) 
at org/xnio/nio/NioTcpChannel. (NioTcpChannel.java:72(Compiled Code)) 
at org/xnio/nio/NioXnioWorker.connectTcpStream(NioXnioWorker.java:313(Compiled Code)) 
at org/xnio/XnioWorker.connectStream(XnioWorker.java:179(Compiled Code)) 
at org/jboss/remoting3/remote/RemoteConnectionProvider.connect(RemoteConnectionProvider.java:214(Compiled Code)) 
at org/jboss/remoting3/EndpointImpl.doConnect(EndpointImpl.java:296(Compiled Code)) 
at org/jboss/remoting3/EndpointImpl.connect(EndpointImpl.java:386(Compiled Code)) 
at org/jboss/ejb/client/remoting/NetworkUtil.connect(NetworkUtil.java:152(Compiled Code)) 
at org/jboss/ejb/client/remoting/NetworkUtil.connect(NetworkUtil.java:133(Compiled Code)) 
at org/jboss/ejb/client/remoting/ConnectionPool.getConnection(ConnectionPool.java:75(Compiled Code)) 
at org/jboss/ejb/client/remoting/RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51(Compiled Code)) 
at org/jboss/ejb/client/remoting/RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:416(Compiled Code)) 
at org/jboss/ejb/client/ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:390(Compiled Code)) 
at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314(Compiled Code)) 
at java/util/concurrent/FutureTask.run(FutureTask.java:149(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908(Compiled Code)) 
at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931(Compiled Code)) 
at java/lang/Thread.run(Thread.java:773(Compiled Code)) 

• A javacore is created as a result of OutOfMemoryError due to native memory exhaustion:

1TISIGINFO     Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "native memory exhausted" received

Resolution

Possible resolutions:

• Don't use scoped context if possible. See How to configure an EJB client in JBoss EAP 6.
• Ty to reduce the number of connections each client makes if they list multiple nodes.

Root Cause

JBoss client creates many remoting threads:
https://access.redhat.com/solutions/1309203

Diagnostic Steps

• Why is scoped context being used?
• Are connections being closed correctly?

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.