How to make EJB's Clustered in JBoss EAP 6?

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.x

Issue

  • How to configure a clustered EJB in JBoss EAP 6?
  • How to invoke a clustered EJB with a remote client?
  • Are additional cluster members automatically detected if EJB's are invoked from a remote client?
  • How to deploy an EJB application in clustered mode?
  • How to create EJB Session Replication application in JBoss EAP 6?
  • We have two JBoss EAP 6.2.4.GA servers in standalone-full mode (server1 and server2). Both servers expose an EJB.

We have another JBoss EAP 6.3.0.GA server in standalone-full mode that use EJB Client API to consume the EJB from server1 and server2.

Everything works fine. The calls to the EJB are balanced.

While We are constantly calling the EJB We restart the server1 (or the server2) and It is never called again.

What should be done to make server1 (or server2) as an eligible node again when the client calls the EJB?

Our jboss-ejb-client.xml contains:

<jboss-ejb-client xmlns="urn:jboss:ejb-client:1.2">
    <client-context invocation-timeout="5000">
        <ejb-receivers exclude-local-receiver="true">
            <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-server1" connect-timeout="7000">
            </remoting-ejb-receiver>
            <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection-server2" connect-timeout="7000">
            </remoting-ejb-receiver>
        </ejb-receivers>
    </client-context>
</jboss-ejb-client>

Resolution

  • EJBs are not clustered by default. To have loadbalancing and failover features, if a standalone client invokes EJB's, it is necessary to mark the application as clustered and configure the server and client appropriately.

  • This can be done by adding the annotation @org.jboss.ejb3.annotation.Clustered to the EJB class or additional configuration in the jboss-ejb3.xml deployment descriptor.

Mark EJB as clustered

  1. In the ejbjar/META_INF/jboss-ejb3.xml deployment descriptor mark all EJB's as clustered :
<jboss:ejb-jar xmlns:jboss="http://www.jboss.com/xml/ns/javaee"
                  xmlns="http://java.sun.com/xml/ns/javaee"
                  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
                  xmlns:s="urn:security"
                  xmlns:c="urn:clustering:1.0"
                  xsi:schemaLocation="http://www.jboss.com/xml/ns/javaee http://www.jboss.org/j2ee/schema/jboss-ejb3-2_0.xsd
                     http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/ejb-jar_3_1.xsd"
                  version="3.1"
                  impl-version="2.0">
    <enterprise-beans>
    </enterprise-beans>
	  <assembly-descriptor>
      <!-- mark all EJB's of the application as clustered (without using the jboss specific @Clustered annotation for each class) -->
      <c:clustering>
          <ejb-name>*</ejb-name>
          <c:clustered>true</c:clustered>
      </c:clustering>
	  </assembly-descriptor>
</jboss:ejb-jar>
  1. Use the annotation @Clustered for EJB3.x :
import org.jboss.ejb3.annotation.Clustered;
import javax.ejb.Stateful;

@Stateful
...
@Clustered
public class MyClusteredEjb implements MyClusteredEjbRemote {
        ....
}
  • Deploy the application to the servers of the cluster.

Note : If both the options are used, then jboss-ejb3.xml deployment descriptor will take precedence.

Create a jboss-ejb-client.xml (If the client is running in EAP 6, the xml goes in the META-INF of the top level deployment)

<jboss-ejb-client xmlns="urn:jboss:ejb-client:1.2">
    <client-context deployment-node-selector="org.jboss.as.test.cluster.CustomDeploymentNodeSelector">
        <ejb-receivers exclude-local-receiver="true">
            <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection"/>
        </ejb-receivers>
        <clusters>
            <cluster name="ejb" max-allowed-connected-nodes="20" cluster-node-selector="org.jboss.as.test.cluster.CustomDeploymentNodeSelector"
                     connect-timeout="15000" username="user1" security-realm="PasswordRealm">
                <connection-creation-options>
                    <property name="org.xnio.Options.SSL_ENABLED" value="false"/>
                    <property name="org.xnio.Options.SASL_POLICY_NOANONYMOUS" value="true"/>
                </connection-creation-options>
            </cluster>
        </clusters>
    </client-context>
</jboss-ejb-client>

For HA on the initial lookup, you just need to add more remoting-ejb-receiver in the standalone*.xml, then just list them in the jboss-ejb-client.xml such as adding remote-ejb-connection2: 

<jboss-ejb-client xmlns="urn:jboss:ejb-client:1.2">
    <client-context deployment-node-selector="org.jboss.as.test.cluster.CustomDeploymentNodeSelector">
        <ejb-receivers exclude-local-receiver="true">
            <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection"/>
            <remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection2"/>
        </ejb-receivers>
        <clusters>
            <cluster name="ejb" max-allowed-connected-nodes="20" cluster-node-selector="org.jboss.as.test.cluster.CustomDeploymentNodeSelector"
                     connect-timeout="15000" username="user1" security-realm="PasswordRealm">
                <connection-creation-options>
                    <property name="org.xnio.Options.SSL_ENABLED" value="false"/>
                    <property name="org.xnio.Options.SASL_POLICY_NOANONYMOUS" value="true"/>
                </connection-creation-options>
            </cluster>
        </clusters>
    </client-context>
</jboss-ejb-client>

Create a jboss-ejb-client.properties (If the client is a standalone client not running in EAP 6)

  • The file is used to specify the ejb-client environment.
# global options
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false

# list of initial server connections
# One server might enough, but for initial failover more are recommended
remote.connections=one,two

remote.connection.one.host=host1
remote.connection.one.port=4447
remote.connection.one.username=ejbone
remote.connection.one.password=password
remote.connection.one.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.one.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false
remote.connection.two.host=host2
remote.connection.two.port=4447
remote.connection.two.username=ejbone
remote.connection.two.password=password
remote.connection.two.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.two.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false

# definition of the cluster
# the name 'ejb' is related to the server configuration
remote.clusters=ejb
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false
remote.cluster.ejb.username=ejbone
remote.cluster.ejb.password=password

NOTE:

  • To be fail safe, during the client startup, it is necessary to use a minimum of two servers in the jboss-ejb-client.properties so that the initial server is not the single point of failure.
    If the client is started and has received the cluster view the client will continue work even if the initial host is not longer available.
  • Make sure that jboss-ejb-client.properties file and jboss-client.jar found in directory $JBOSS_HOME/bin/client are the in the client CLASSPATH.

Configure standalone client

  • In your Client code configure the look up as follows :
Properties props = new Properties();
props.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
InitialContext ic = new InitialContext(props);
myEJB = ic.lookup("ejb:<app name>/<module name>/<bean name>!<full qualified view name>")
  • The lookup depends on the application and it is possible to see the JNDI name in the logfile of the server if the application is deployed.
  • The proxy, here 'myEJB' must be reused to get the full functionality, see this article

Quickstart

  • It is possible to use each Stateless Session Bean to check the behavior for this test. As a simple example the Content from github.com is not included.ejb-remote quickstart can be used.
  • In addition, there is a quickstart with a clustered application available: Content from github.com is not included.ejb-multi-server.
  • If a quickstart is used follow the provided instructions.
  • Start two or more servers using a HA profile, i.e. bin/standalone.sh -c standalone-ha.xml.
  • Ensure that the remoting security is disabled or a user is added for the ApplicationRealm

See also:
This content is not included.JBoss EAP 6.4 Development Guide

Diagnostic Steps

  • Lets suppose there is an EJB 3.x with @Clustered annotation deployed on the cluster of two standalone servers.
  • Using the attached testcase, the file named testcase.zip contains the following:
    • MyClusteredEJBClient.java which has the properties specified programatically in the client which is another option to specifying a separate ejb client properties file.
    • MyClusteredEJB.jar
    • standalone-ha.xml of both the servers.
    • userrolesproperties consist of two properties file where user credentials and user to role mapping is configured.

The loadbalancing and the failover can be tested in the following way:

  • Start the two standalone servers in the following way:
./standalone.sh -c standalone-ha.xml -Djboss.node.name=server-1 -Djboss.server.base.dir=../standalone_1
./standalone.sh -c standalone-ha.xml -b 10.10.10.150 -bmanagement 10.10.10.150 -Djboss.node.name=server2 -Djboss.server.base.dir=../standalone_2

**NOTE ** : copied $JBOSS_HOME standalone folder to create other two folders one being $JBOSS_HOME/standalone1 and other being $JBOSS_HOME/standalone2, where JBOSS_HOME is the same.

  • Deploy the EJB on both servers and the servers will able to join the cluster as follows:

    • server-2:
    21:02:37,289 INFO  [org.jboss.as.clustering] (MSC service thread 1-8) JBAS010238: Number of cluster members: 1
    • server-1:
    21:02:48,712 INFO  [org.jboss.as.clustering] (MSC service thread 1-6) JBAS010238: Number of cluster members: 2

  • Client and EJB communication:

    • Start the two standalone EJB clients i.e ran MyClusteredEJBClient.java as two different java processes.
    • When client-1 connects the following is seen in the server-2 logs:
    00:02:07,918 INFO  [stdout] (EJB default - 2) Hello Clustered EJBhas the hashcode 1425160101
    • Here hashCode() is used in this example as it is unique for all clients.
    • When client-2 connected the following was seen in the server-1 logs:
    00:02:14,632 INFO  [stdout] (EJB default - 2) Hello Clustered EJBhas the hashcode 1724681368
    • This shows that there is a load-balancing happening where 2 different clients went on two different servers to connect to the statefull-clustered EJB.
    • Now the client-1 and client-2 will go on 30 seconds sleep with Thread.sleep(30000) specified in clients.
    • The server-1 is brought down to check whether the failover happens when client-2 which was connected to server-1 found out that the server-1 is dead.
    • As seen below the failover happened successfully when client-2 recieved same hashcode from server-2 remote EJB as follows:-
        Lookup successful
    Hello Clustered EJBhas the hashcode 1724681368
    Jun 11, 2013 12:02:30 AM org.jboss.ejb.client.remoting.ChannelAssociation$ResponseReceiver handleEnd
    NFO: EJBCLIENT000016: Channel Channel ID ca9f886e (outbound) of Remoting connection 756469e8 to localhost/127.0.0.1:4447 can no longer process messages

    <here the client goes to sleep where the server-1 was brought down and then it recieves the same hashcode from the server-2 as seen below>

    
    1724681368
    1724681368
    1724681368
    • As you can see the same hashcode is received from the server-2 when client invokes remote.getEcho() as specified in MyClusteredEJBClient.java after the sleep was called and at that time the server was brought down to which client-1 was connected.

  • Hence as we see the loadbalancing happens as well as failover happens successfully.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.