The EJB invocations are not well balanced across my clustered EAP6 servers

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 6.x

Issue

• Why I see an uneven loadbalancing for remote EJB invocations to the EAP cluster?
• How can I check the node selection for the DeploymentNodeSelector or ClusterNodeSelector?
• ClusterNodeSelector not getting called from client, what it the reason?
• The DeploymentNodeSelector works well and the ClusterNodeSelectordoes not, why?
• Need help to understand behaviour of EJB clients for clustered applications

Resolution

To have the best performance the client should initialize the ejb-client only once. This can be done by using the jboss-ejb-client.properties or the JBoss specific API, see Configure EJB client for more information.
After the client has been initialized the proxy can be retrieved by a lookup(...).
This proxy is created on the client side based on the given BusinessInterface, the server is NOT connected at this state. The proxy does not have any information about the server environment.
After this the process followed is:

• First invocation
  • will use the DeploymentNodeSelector to select the server which has the appropriate EJB
  • The server might return a cluster-view if this is the first invocation in this JVM
  • The invocation is processed
  • The proxy will get an affinity to the cluster
• Further invocations
  • The cluster affinity provide the name of the cluster
  • The ClusterNodeSelector is used with a list of all cluster members to select a node
  • The invocation is processed

As a consequence of this implementation the load-balancing will not optimal if the client do a new lookup for every invocation. In this case the invocation will not use the whole cluster for the invocation but the list of initial nodes. Also there is no smooth failover as there is no list of all possible nodes if the initial one failed during runtime.

The load-balancing for non-clustered and clustered invocation is Random by default which provide a good policy, it can be customized if needed see How to configure clustered EJB load balancing for more informations

Root Cause

Depend on the configuration and client side approach the cluster might not detected correctly or the client is not able to use it.

Since EAP6 the new ejb-client library will take care of the EJB invocations.
The logic is inside the client-library and the client control the distribution. This is different from former versions as here the server is connected after the lookup to provide the proxy.

Diagnostic Steps

• The EJB proxy must be cached and re-used to have benefit from each cluster update if server nodes are leaving or joining
• To track the node selection the attached EJBClient.NodeSelectorTracking.btm script for Content from byteman.jboss.org is not included.Byteman can be used.
  Using Byteman to troubleshoot Java issues
  If unsure how to use byteman or have concerns please don't hesitate to open a case

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.