java.nio.channels.ClosedChannelException is thrown for remote EJB calls that take longer than 60 seconds

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 7.4

Issue

  • If a remote EJB call takes longer than 60 seconds, the TCP connection is closed from server-side when 60 seconds have elapsed since the start of the remote EJB request. We can see the following stack trace in the remote EJB client-side and no response is returned from server-side.
Exception in thread "main" javax.ejb.EJBException: java.nio.channels.ClosedChannelException
	at org.jboss.ejb.protocol.remote.EJBClientChannel$MethodInvocation.handleClosed(EJBClientChannel.java:1287)
	at org.jboss.remoting3.util.InvocationTracker.connectionClosed(InvocationTracker.java:222)
	at org.jboss.remoting3.util.InvocationTracker.lambda$new$0(InvocationTracker.java:70)
	at org.jboss.remoting3.spi.SpiUtils.safeHandleClose(SpiUtils.java:50)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable$CloseHandlerTask.run(AbstractHandleableCloseable.java:520)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.runCloseTask(AbstractHandleableCloseable.java:425)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeComplete(AbstractHandleableCloseable.java:286)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.closeAction(RemoteConnectionChannel.java:510)
	at org.jboss.remoting3.spi.AbstractHandleableCloseable.closeAsync(AbstractHandleableCloseable.java:368)
	at org.jboss.remoting3.remote.RemoteConnectionHandler.closeAllChannels(RemoteConnectionHandler.java:623)
	at org.jboss.remoting3.remote.RemoteConnectionHandler.receiveCloseRequest(RemoteConnectionHandler.java:226)
	at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:97)
	at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:51)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
	at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
Caused by: java.nio.channels.ClosedChannelException
	... 17 more
  • The undertow http-listener has write-timeout attribute in standalone.xml. If write-timeout is set to undefined same as default, this issue is no longer reproduced]
<http-listener name="default" write-timeout="60000" socket-binding="http" redirect-socket="https" enable-http2="true"/>

Resolution

This is a known issue This content is not included.JBEAP-24842, and the fix is included in EAP 7.4.13.GA.

As a workaround, set heartbeat-interval to a value shorter than undertow's write-timeout in src/main/resources/wildfly-config.xml on standalone EJB client side. If the write-timeout is set to 60000 ms in http-listener, set heartbeat-interval="15000". The default heartbeat-interval is 60000 ms, introduced by This content is not included.JBEAP-16805 fixed in JBoss EAP 7.2.3.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <jboss-ejb-client xmlns="urn:jboss:wildfly-client-ejb:3.0">
        <connections>
            <connection uri="remote+http://127.0.0.1:8080"/>
        </connections>
    </jboss-ejb-client>
    <endpoint xmlns="urn:jboss-remoting:5.1" heartbeat-interval="15000">
        <connections/>
    </endpoint>
</configuration>

NOTE: If the standalone EJB client is implemented with property-based setting as follows without wildfly-config.xml, the heartbeat interval setting is not available. This content is not included.WFNC-67 is filed as a Request For Enhancement.

Properties props = new Properties();
props.put(Context.INITIAL_CONTEXT_FACTORY,  "org.wildfly.naming.client.WildFlyInitialContextFactory");
props.put(Context.PROVIDER_URL, "remote+http://127.0.0.1:8080");
Context context = new InitialContext(props);

Root Cause

  • In jboss-remoting which is used as remote EJB's protocol implementation, HEARTBEAT messages are sent from ejb-client to ejb-server at 60-second intervals by default.
  • This issue causes undertow's write-timeout timer is not reset when the entire remote EJB requests message is received on server-side. When a server receives the first HEARTBEAT from ejb-client 60 seconds after receiving the entire request message, undertow reaches write-timeout at the time of writing ack to HEARTBEAT to the ejb-client, then closes the TCP connection from server-side, before returning a response for remote ejb call.

Diagnostic Steps

When this issue is reproduced, the following WARN message is logged in server.log on remote EJB server-side. The following message indicates that the remote EJB response cannot be written to the socket because the TCP connection has been closed due to write-timeout.

10:43:33,621 WARN  [org.jboss.ejb.client.remoting] (default task-1) EJBCLIENT000519: Exception occurred when writing EJB response to invocation 32912 over channel Channel ID 73fcaf2c (inbound) of Remoting connection 2c649192 to <host-name>/127.0.0.1:33286 of endpoint "<node-name>" <10b2bfa6>: org.jboss.remoting3.NotOpenException: Writes closed
	at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:108)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:293)
	at org.jboss.remoting3.util.MessageTracker.writeMessage(MessageTracker.java:98)
	at org.jboss.remoting3.util.MessageTracker.openMessageUninterruptibly(MessageTracker.java:90)
	at org.jboss.ejb.protocol.remote.EJBServerChannel$RemotingInvocationRequest$1.writeInvocationResult(EJBServerChannel.java:959)
	at org.jboss.as.ejb3.remote.AssociationImpl.lambda$receiveInvocationRequest$0(AssociationImpl.java:292)
	at org.jboss.as.ejb3.remote.AssociationImpl.execute(AssociationImpl.java:345)
	at org.jboss.as.ejb3.remote.AssociationImpl.receiveInvocationRequest(AssociationImpl.java:298)
	at org.jboss.ejb.protocol.remote.EJBServerChannel$ReceiverImpl.handleInvocationRequest(EJBServerChannel.java:477)
	at org.jboss.ejb.protocol.remote.EJBServerChannel$ReceiverImpl.handleMessage(EJBServerChannel.java:212)
	at org.jboss.remoting3.remote.RemoteConnectionChannel.lambda$handleMessageData$3(RemoteConnectionChannel.java:432)
	at org.jboss.remoting3.EndpointImpl$TrackingExecutor.lambda$execute$0(EndpointImpl.java:993)
	at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
	at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.lang.Thread.run(Thread.java:750)
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.