EJB invocation gets failed with error "Connection reset by peer" in JBoss EAP 6

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.x

Issue

  • Is it possible to keep a ejb-client connection alive if there is a firewall or network configuration which drop a connection on inactivity?
  • Following errors are seen in the server logs
04:27:20,220 ERROR [org.jboss.remoting.remote.connection] (Remoting "node1" read-1) JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
  • EJB2 Stateless session beans are deployed on JBoss 6 EAP whose remote view is being consumed/invoked by web layer with the use standard ServiceLocator pattern implementation. Service Locator class also cache remote home proxy and InitialContext object (i.e. object which is used to locate that EJBHomeProxy). The EJB invocation fails frequently after some time of normal running:
TRACE 22:12:37,747 Remoting "client-endpoint" read-1 RemoteReadListener - Connection error detail java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(Unknown Source)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
        at sun.nio.ch.IOUtil.read(Unknown Source)
        at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
        at org.xnio.nio.AbstractNioStreamChannel.read(AbstractNioStreamChannel.java:249)
        at org.xnio.channels.FramedMessageChannel.receive(FramedMessageChannel.java:87)
        at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:72)
        at org.jboss.remoting3.remote.RemoteReadListener.handleEvent(RemoteReadListener.java:45)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
        at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
        at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
        at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
        at org.xnio.nio.NioHandle.run(NioHandle.java:90)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:187)
  • A connection for EJB3 invocation initial works. But from time to time, it looks like around the same hours, the invocations failed.

Resolution

There are properties to set a HEARTBEAT to keep connection alive.
The heartbeat setting should be lower than any idle-timeout settings on network hardware (e.g. F5 Load Balancer, firewall, router etc.) in order to keep the connection open. Setting heartbeat too low might result in unnecessary network traffic.

Note: If the heartbeat is enabled on both the server & client side, the heartbeat will be sent from the side that has the lower HEARTBEAT_INTERVAL setting as it will run first and reset the heartbeat timer.

Note It is recommended the READ_TIMEOUT be at least 2 * HEARTBEAT_INTERVAL

  • If the heartbeat is not fast enough, then the firewall times out the connection, it will cause the read timeout, then writes closed.
  • If the READ_TIMEOUT is less than the heartbeat, then the connection will get killed off before the heartbeat gets sent.
  • Both of these are fixed by setting the READ_TIMEOUT to be a multiple of the HEARTBEAT_INTERVAL. 
org.xnio.Options.READ_TIMEOUT

Enabling Heartbeat server side

The heartbeat can be enabled sever side via the remoting subsystem. This will send a heartbeat from the server to any connections made on the remoting connector.

        <subsystem xmlns="urn:jboss:domain:remoting:1.1">
            <connector name="remoting-connector" socket-binding="remoting" security-realm="ApplicationRealm">
              <properties>
                <property name="org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL" value="30000"/>
                <property name="KEEP_ALIVE" value="true"/>
                <property name="READ_TIMEOUT" value="60000"/>
              </properties>
            </connector>
        </subsystem>

Enabling Heartbeat client side

The heartbeat can also be enabled on the client side to send heartbeats to the server.

If a standalone client with jboss-ejb-client.properties is used

Add the following properties to the relevant connection(s)

remote.connections=default
...
remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL=<value in milliseconds>
remote.connection.default.connect.options.org.xnio.Options.KEEP_ALIVE=true

If the client connect a cluster this need to be configured in the properties, see this article, and add the heartbeat property for the cluster as well

remote.cluster.ejb.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL=<value in milliseconds>

If a server to server connection is used (deployed EJB application)

If the EJB is invoked inside on a server with an outbound-connection configuration, this configuration need to be adjusted to:

<subsystem xmlns="urn:jboss:domain:remoting:1.1">
    ...
    <outbound-connections>
        <remote-outbound-connection name="remote-ejb-connection-1" ...>
            <properties>
                <property name="SASL_POLICY_NOANONYMOUS" value="false"/>
                <property name="SSL_ENABLED" value="false"/>
                <property name="KEEP_ALIVE" value="true"/>
                <property name="READ_TIMEOUT" value="<value in milliseconds>"/>
               <property name="org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL" value="<value in milliseconds>"/>
            </properties>
        </remote-outbound-connection>
        ....
    </outbound-connections>
    ....
</subsystem>

The configuration applies for standalone.xml or domain.xml as well, for domain configuration you need to choose the correct profile.

If the target is a cluster the jboss-ejb-client.xml descriptor inside the application needs to be adjusted accordingly

<jboss-ejb-client ...>
    <client-context>
       <ejb-receivers>
            <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-1" />
        </ejb-receivers>
        <clusters>
            <cluster name="ejb" ...>
                <connection-creation-options>
                    ...
                    <property name="SSL_ENABLED" value="false"/>
                    <property name="KEEP_ALIVE" value="true"/>
                    <property name="READ_TIMEOUT" value="<value in milliseconds>"/>
                    <property name="org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL" value="<value in milliseconds>"></property>
                </connection-creation-options>
            </cluster>
        </clusters>
    </client-context>
</jboss-ejb-client>

If the Remote naming approach is used

Add the following properties to the InitialContext:

Properties p = new Properties();
p.put(javax.naming.Context.PROVIDER_URL, "remote://<host>:4447");
  // other propeties
  // add this
p.put("jboss.naming.client.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL", "<value in milliseconds>");
p.put("jboss.naming.client.connect.options.org.xnio.Options.KEEP_ALIVE", "true");
p.put("jboss.naming.client.connect.options.org.xnio.Options.READ_TIMEOUT", "<value in milliseconds>");

InitialContext ic = new InitialContext(p);

Consider migrate to the ejb-client approach as it is not recommended to use remote-naming for EJB invocations.

If EAP 6.0.x is used

If EAP 6.0.x is used we recommend upgrading it to EAP 6.2 as soon as possible because there were various EJB related bug fixes.
For EAP 6.0.1 there is an one off patch which has some of the bug fixes. This might help until an upgrade is possible. See This content is not included.Fix various EJB issues for download.

If EAP 6.1 is used

Consider upgrade to EAP 6.2 or newer to have the latest EJB client bug fixes.

If EAP 6.4 is used

Remoting has been upgraded for later versions of EAP 6.x. If 6.4 is in use (as with the Fuse product line), please ensure the schema in use is jboss-as-remoting_1_2.xsd. The server configuration should reference this as follows:

<subsystem xmlns="urn:jboss:domain:remoting:1.2">

Root Cause

"Connection reset by peer" suggests the TCP connection was broken by the other end or something in the middle of the network.

There is often network hardware, like F5 load balancer, firewall, router or a L3 switch, which have an idle-timeout for connections that might reset the connection if idle for a period of time.

Diagnostic Steps

Check following items to find out the root cause:

  • Does this happen after a particular time period since restart the service / last successful invocation?
    For example, "it always does work after restart the server, but will have this error after running for *** minutes"
  • Does this occur after a period of not much activity?
    For example, "it always happens at the beginning of the day, or after lunch break"
ERROR 04:24:09,708 Remoting "config-based-naming-client-endpoint" read-1 RemoteConnection - JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
ERROR 04:52:19,468 Remoting "config-based-naming-client-endpoint" read-1 RemoteConnection - JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
ERROR 05:22:01,247 Remoting "config-based-naming-client-endpoint" read-1 RemoteConnection - JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
ERROR 05:41:36,507 Remoting "config-based-naming-client-endpoint" read-1 RemoteConnection - JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
ERROR 06:41:40,058 Remoting "config-based-naming-client-endpoint" read-1 RemoteConnection - JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
ERROR 07:25:26,928 Remoting "config-based-naming-client-endpoint" read-1 RemoteConnection - JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
ERROR 07:52:23,637 Remoting "config-based-naming-client-endpoint" read-1 RemoteConnection - JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
ERROR 08:02:47,188 Remoting "config-based-naming-client-endpoint" read-1 RemoteConnection - JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
ERROR 21:12:49,698 Remoting "config-based-naming-client-endpoint" read-1 RemoteConnection - JBREM000200: Remote connection failed: java.io.IOException: Connection reset by peer
  • Is there any firewall or network device configured between the EJB client and the EJB server?
    Are there "Idle timeout" configurations on these devices?
  • Is the network environment stable? Is there any possibility that the network connection could be reset in some cases?
  • Are EJB client and EJB server in the same network?
    Is there a network hardware like firewall or router in between?

related community issues
Content from community.jboss.org is not included.Remote method call fails after 30 minutes

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.