Heartbeat on EJB connections in JBoss EAP 7.1+

Solution Verified - Updated 14 Jun 2024

Environment

Red Hat JBoss Enterprise Application Platform (EAP)
- 7.1
- 7.2
Enterprise Java Beans (EJB)
Remoting/HTTP Remoting

Issue

Enable heartbeat on connections

Resolution

For CLI commands, the commands applies for standalone.xml or domain.xml, but for domain configuration you need to choose the correct profile by prefixing the commands with /profile=MYPROFILE/.

Enabling Server Heartbeat

The heartbeat can be enabled sever side via the remoting subsystem. This will send a heartbeat from the server to any connections made on the http-remoting-connector:

        /subsystem=remoting/http-connector=http-remoting-connector/property=org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL:add(value=30000)
        /subsystem=remoting/http-connector=http-remoting-connector/property=KEEP_ALIVE:add(value=true)
        /subsystem=remoting/http-connector=http-remoting-connector/property=READ_TIMEOUT:add(value=60000)

Right after the commands are applied, the outcome should be success.
Change the value with write-value command on the connector (example below is http-connector on the remoting subsystem):

        /subsystem=remoting/http-connector=http-remoting-connector/property=READ_TIMEOUT:write-value(name=value, value=60000)

Reading the attribute value:

$ /subsystem=remoting/http-connector=http-remoting-connector/property=READ_TIMEOUT:read-attribute(include-defaults=true,name=value)
{
    "outcome" => "success",
    "result" => "60000"
}

The resulting XML will be:

        <subsystem xmlns="urn:jboss:domain:remoting:4.0">
            <http-connector name="http-remoting-connector" connector-ref="default" security-realm="ApplicationRealm"><!-- on the remoting subsystem and http-remoting-connector -->
                <properties>
                    <property name="org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL" value="30000"/>
                    <property name="KEEP_ALIVE" value="true"/>
                    <property name="READ_TIMEOUT" value="60000"/>
                </properties>
            </http-connector>
        </subsystem>

Enabling Client Heartbeat

The heartbeat can also be enabled on the client side to send heartbeats to the server.

Server to server connections (deployed EJB application)
If the EJB is invoked inside on a server with an outbound-connection configuration, this configuration need to be adjusted to

            /subsystem=remoting/outbound-connection=remote-ejb-connection-1/property=KEEP_ALIVE:add(value=true)
            /subsystem=remoting/outbound-connection=remote-ejb-connection-1/property=READ_TIMEOUT:add(value=60000)
            /subsystem=remoting/outbound-connection=remote-ejb-connection-1/property=org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL:add(value=30000)

Change the value with write-value command (example below sets the property KEEP_ALIVE on the remoting subsystem):

            /subsystem=remoting/outbound-connection=remote-ejb-connection-1/property=KEEP_ALIVE:write-value(name=value, value=true)

The corresponding XML would look like this

            <subsystem xmlns="urn:jboss:domain:remoting:1.1">
                ....
                <outbound-connections>
                    <remote-outbound-connection name="remote-ejb-connection-1" ...>
                        <properties>
                            ....
                            <property name="KEEP_ALIVE" value="true"/>
                            <property name="READ_TIMEOUT" value="<value in milliseconds>"/>
                            <property name="org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL" value="<value in milliseconds>"/>
                        </properties>
                    </remote-outbound-connection>
                    ....
                </outbound-connections>
                ....
            </subsystem>

Target is a cluster the jboss-ejb-client.xml descriptor inside the application needs to be adjusted accordingly:

            <jboss-ejb-client ...>
                <client-context>
                <ejb-receivers>
                        <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-1" />
                    </ejb-receivers>
                    <clusters>
                        <cluster name="ejb" ...>
                            <connection-creation-options>
                                ...
                                <property name="SSL_ENABLED" value="false"/>
                                <property name="KEEP_ALIVE" value="true"/>
                                <property name="READ_TIMEOUT" value="<value in milliseconds>"/>
                                <property name="org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL" value="<value in milliseconds>"></property>
                            </connection-creation-options>
                        </cluster>
                    </clusters>
                </client-context>
            </jboss-ejb-client>

Standalone client with wildfly-config.xml is used:
read-timeout should be 2 x heartbeat-interval , read-timeout & write-timeout timeouts are in milliseconds and heartbeat-interval is in milliseconds

            <?xml version="1.0" encoding="UTF-8"?>
            <configuration>
                <endpoint xmlns="urn:jboss-remoting:5.0">
                    <connections>
                        <connection destination="remote+http://host1:8080" read-timeout="60000" write-timeout="60000" heartbeat-interval="30000"/>
                        <connection destination="remote+http://host2:8080" read-timeout="60000" write-timeout="60000" heartbeat-interval="30000"/>
                    </connections>
                </endpoint>
            </configuration>

NOTE: There have been issues with this configuration which have been fixed with JBoss EAP 7.2 CP2, or later

Standalone client with jboss-ejb-client.properties
Add the following properties to the relevant connection(s):

            remote.connections=default
            ...
            remote.connection.default.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL=<value in milliseconds>
            remote.connection.default.connect.options.org.xnio.Options.KEEP_ALIVE=true

Cluster connection
If the client connect a cluster this need to be configured in the properties, see this article, and add the heartbeat property for the cluster as well

            remote.cluster.ejb.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL=<value in milliseconds>

Remote naming approach (Not Recommended)
Add the following properties to the InitialContext:

            Properties p = new Properties();
            p.put(javax.naming.Context.PROVIDER_URL, "remote+http://<host>:4447");
            // other propeties
            // add this
            p.put("jboss.naming.client.connect.options.org.jboss.remoting3.RemotingOptions.HEARTBEAT_INTERVAL", "<value in milliseconds>");
            p.put("jboss.naming.client.connect.options.org.xnio.Options.KEEP_ALIVE", "true");
            p.put("jboss.naming.client.connect.options.org.xnio.Options.READ_TIMEOUT", "<value in milliseconds>");

            InitialContext ic = new InitialContext(p);

Consider migrating to the ejb-client approach as it is not recommended to use remote-naming for EJB invocations.

Root Cause

There are properties to set a HEARTBEAT to keep connection alive.
The heartbeat setting should be lower than any idle-timeout settings on network hardware (e.g. F5 Load Balancer, firewall, router etc.) in order to keep the connection open. Setting heartbeat too low might result in unnecessary network traffic.

Recommendations

If the heartbeat is enabled on both the server & client side, the heartbeat will be sent from the side that has the lower HEARTBEAT_INTERVAL setting as it will run first and reset the heartbeat timer.
It is recommended the READ_TIMEOUT be at least 2 * HEARTBEAT_INTERVAL
If the heartbeat is not fast enough, then the firewall times out the connection, it will cause the read timeout, then writes closed.
If the READ_TIMEOUT is less than the heartbeat, then the connection will get killed off before the heartbeat gets sent.
Both of these are fixed by setting the READ_TIMEOUT to be a multiple of the HEARTBEAT_INTERVAL.
```
  org.xnio.Options.READ_TIMEOUT
```

Related Solutions

Solution	Usage
EJB invocation gets failed with error "Connection reset by peer" in JBoss EAP 6 / 7	For connection reset issues
Content from docs.jboss.org is not included.Remote EJB invocations via JNDI - EJB client API or remote-naming project	for more details about "EJB client" and "Remote naming"
How to configure an EJB client in JBoss EAP 6	to learn how to implement an "EJB client" on EAP 6 platform
This content is not included.7.5.6. EJB Client Properties	for available EJB client properties

Diagnostic Steps

JBoss Remoting Logging at TRACE will show log messages when the heartbeat is sent Content from issues.jboss.org is not included.REM3-255

/subsystem=logging/logger=org.jboss.remoting.remote:add(level=FINEST)

The log messages are: Sending connection alive and Sending connection alive ack

SBR

JBoss Base AS

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Learn more

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.