How to configure clustered EJB load balancing policy in EAP 6

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 6.x
    • 7.0.x

Issue

  • What is the correct way of changing the EJB load balancing policy? This was using the Clustered interface's loadBalancePolicy attribute in EAP 5.
  • What cluster load balancing policy is used for EJB invocations in EAP6 by default
  • How the Selector decides when to open a connection with a node in the cluster if already it has not been opened? Does it do it on the basis of load?
  • We have a >20 node cluster and a high amount of requests per second, would it be necessary for us to override the default implementation and have a different implementation used?
  • As of now, the cluster node selector is set to "org.jboss.ejb.client.RandomClusterNodeSelector", how can I make it do a round robin node selector.
  • What EJB load balancing policies are available in EAP 6?
  • Is the random load balancing policy the only implementation? What happened to RoundRobin?
  • Each EJB invocation will stay local if the requested EJB is locally deployed, can I change that behaviour?

Resolution

There are two types of NodeSelectors, one for not clustered EJB's and one for clustered EJB's.
By default, if there is no custom configuration of such selectors, the default implementation provided by the ejb-client library is used which is a RandomRobin policy. As this policy provide a similar load distribution as RoundRobin. The Random policy is the only provided at this time and there are a RandomDeploymentNodeSelector and a ClusterNodeSelector available.
If the client is inside of an EAP instance there is a special implementation to prefer the local instance for invocations to prevent from unnecessary cross-server invocations.

The procedure for invocation is as followed

  • after client initialization the first invocation of an EJB will use the DeploymentNodeSelector to route to an appropriate server instance for the application context (app-name/modul-name) by using the configured connections
  • if there is no cluster context returned (async to the invocation) the DeploymentNodeSelector is used for further invocations
  • if the server send a cluster-view the client will establish a context for this cluster and add connections to the nodes of the cluster
    • all invocations onwards will use the ClusterNodeSelector for all EJB's which are aligned to this cluster

In any case the Selector itself must not open connections.

When should a customized NodeSelector be implemented?

By default the policy is a random selection of nodes and the local node is prefered if the application is available here.
The default ClusterNodeSelector should be replaced when:

  • a big cluster >20 nodes is used and it is identified that long running use-cases are overwhelming the 20 nodes. The default implementation uses only a maximum of 20 connections, new connections will only be opened if a node fails.
  • to implement a customized balancing policy, i.e. with a load-factor or non-symetric policy
  • it is prefered to use other nodes as well to spread the load - consider this affects the complete deployment and remote invocations are more expensive!

It is not necessary to do so for a RoundRobin policy, the random policy will have the same well-balanced behaviour.

How to implement

A class which implements the ClusterNodeSelector needs to be available on the client application side.

For the selection, the current clusterName is provided.
The String[] availableNodes contains all registered nodes which are able to handle the current EJB invocation.
connectedNodes contains all the already connected nodes and is a subset of availableNodes.

Note: The connections must not be opened inside the Selector implementation! This will be handled automatically.

public class MyClusterNodeSelector implements org.jboss.ejb.client.ClusterNodeSelector {
  @Override
  public String selectNode(final String clusterName, final String[] connectedNodes, final String[] availableNodes) {
    if (availableNodes.length == 1) {
      return availableNodes[0];
    }
    final Random random = new Random();
    final int randomSelection = random.nextInt(availableNodes.length);
    // return a String with one of the node names in connected* or availableNodes
    return availableNodes[randomSelection];
}

Note: There is only one instance of the NodeSelector for different EJB invocations in the same JVM so it might be used for different Cluster's and EJB's invocations and the implementation must be Thread-safe and not store a state.

To activate the NodeSelector the property for the selector must be set with the fully qualified class-name of the NodeSelector implementation.

For a JavaEE application

The outbound connections are configured in the jboss-ejb-client.xml descriptor. The Selector needs to be added to the cluster configuration as follows:

<jboss-ejb-client xmlns:xsi="urn:jboss:ejb-client:1.2">
  <client-context >
    <ejb-receivers >
      ...
    </ejb-receivers>

    <clusters>
      <cluster ... cluster-node-selector="my.package.MyClusterNodeSelector">
      </cluster>
    </clusters>
  </client-context>
</jboss-ejb-client>

Important
If all nodes are part of the same cluster the ejb-receivers element can be completely dropped, only the cluster element with the node-selector attribute is needed. But the connection-creation is necessary to disable the SSL!

For a standalone client with properties

The connections are configured in the jboss-ejb-client.properties file. The Selector needs to be added like this:

remote.connections=default
remote.connection.default.host=localhost
remote.connection....

remote.clusters=ejb
remote.cluster....
remote.cluster.ejb.clusternode.selector=my.package.MyClusterNodeSelector
If the EJBClientContext is set programaticaly
Properties p = new Properties();
p.put("remote.connections", "default");
p.put("remote.connection.default.host", "localhost");
p.put(.......)  // other properties

final String clusterName = "ejb";
p.put("remote.clusters", clusterName);
p.put("remote.cluster."+clusterName+".clusternode.selector", MyClusterNodeSelector.class.getName());

EJBClientConfiguration cc = new PropertiesBasedEJBClientConfiguration(p);
ContextSelector<EJBClientContext> selector = new ConfigBasedEJBClientContextSelector(cc);
EJBClientContext.setSelector(selector);
For a scoped-context client
Properties p = new Properties();
p.put("org.jboss.ejb.client.scoped.context", true);
p.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
p.put("remote.connections", "default");
p.put("remote.connection.default.host", "localhost");
p.put(.......)  // other properties

final String clusterName = "ejb";
p.put("remote.clusters", clusterName);
p.put("remote.cluster."+clusterName+".clusternode.selector", MyClusterNodeSelector.class.getName());

Root Cause

In difference to former EAP versions where the proxy is provided by the server the balancing policy will be now the responsibility of the client side API.
Former implementations like RoundRobin or FirstAvailable are not longer provided.

How to implement a custom loadbalancing policy for EJB calls in EAP6
The EJB invocations are not well balanced across my clustered EAP6 servers

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.