Exception JBREM000200 is thrown at remote-client side if a clustered EJB is called in EAP6
Environment
- Red Hat JBoss Enterprise Application Platform (EAP)
- 6.x
Issue
- After changing a single server, with EJB application, to a cluster, an Exception is logged on the client side but the EJB invocations are working. Is this a wrong configuration?
- There is no LoadBalancing if a client invokes a clustered EJB. Only the initial server is accessed. Why are the other nodes not used as this was done automatically with EAP5 clients?
- How to configure the ejb-client to use a clustered application
- There is an INFO message that the connection can not created to a cluster node. What is the reason for that?
[Server:app-main] INFO [org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager] (ejb-client-cluster-node-connection-creation-4-thread-1) Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='master:app-one', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='127.0.0.1', destinationPort=4547}], resolvedDestination=[Destination address=127.0.0.1, destination port=4547]} in cluster ejb: java.lang.RuntimeException: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
- Why is there an ERROR message (with EAP 6.1.0) if an ejb-client tries to invoke a clustered application?
ERROR org.jboss.remoting.remote.connection> JBREM000200: Remote connection failed: javax.security.sasl.SaslException: Cannot get userid/password [Caused by javax.security.auth.callback.UnsupportedCallbackException]
INFO org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager> Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='torvm-perfamp01:smp_svr1', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='10.10.10.10', destinationPort=10123}], resolvedDestination=[Destination address=10.10.10.10, destination port=10123]} in cluster ejb
java.lang.RuntimeException: javax.security.sasl.SaslException: Cannot get userid/password [Caused by javax.security.auth.callback.UnsupportedCallbackException]
at org.jboss.ejb.client.remoting.IoFutureHelper.get(IoFutureHelper.java:91)
at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:75)
at org.jboss.ejb.client.remoting.RemotingConnectionManager.getConnection(RemotingConnectionManager.java:51)
at org.jboss.ejb.client.remoting.RemotingConnectionClusterNodeManager.getEJBReceiver(RemotingConnectionClusterNodeManager.java:77)
at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:406)
at org.jboss.ejb.client.ClusterContext$EJBReceiverAssociationTask.call(ClusterContext.java:380)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: javax.security.sasl.SaslException: Cannot get userid/password [Caused by javax.security.auth.callback.UnsupportedCallbackException]
at com.sun.security.sasl.ClientFactoryImpl.getUserInfo(ClientFactoryImpl.java:157)
at com.sun.security.sasl.ClientFactoryImpl.createSaslClient(ClientFactoryImpl.java:94)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities$1.run(ClientConnectionOpenListener.java:381)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities$1.run(ClientConnectionOpenListener.java:379)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:379)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:227)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.nio.NioHandle.run(NioHandle.java:90)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:187)
at ...asynchronous invocation...(Unknown Source)
at org.jboss.remoting3.EndpointImpl.doConnect(EndpointImpl.java:270)
at org.jboss.remoting3.EndpointImpl.connect(EndpointImpl.java:386)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:151)
at org.jboss.ejb.client.remoting.NetworkUtil.connect(NetworkUtil.java:132)
at org.jboss.ejb.client.remoting.ConnectionPool.getConnection(ConnectionPool.java:73)
... 9 more
Caused by: javax.security.auth.callback.UnsupportedCallbackException
at org.jboss.ejb.client.DefaultCallbackHandler.handle(DefaultCallbackHandler.java:47)
at com.sun.security.sasl.ClientFactoryImpl.getUserInfo(ClientFactoryImpl.java:136)
at com.sun.security.sasl.ClientFactoryImpl.createSaslClient(ClientFactoryImpl.java:94)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities$1.run(ClientConnectionOpenListener.java:381)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities$1.run(ClientConnectionOpenListener.java:379)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:379)
at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:227)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
at org.xnio.nio.NioHandle.run(NioHandle.java:90)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:187)
Resolution
The jboss-ejb-client.properties, or even jboss-ejb-client.xml for a server-side application, must be expanded to include a cluster configuration as follows:
The cluster name 'ejb' is related to the servers ejb subsystem configuration.
The connection options must be set similar to the options of the initial connection.
If the connection is secured the properties username/password needs to be set also to be able to establish the connection.
Standalone client with jboss-ejb-client.properties configuration
# Declare the cluster(s), ',' separated list
remote.clusters=ejb
# Connection configuration(s) for the "ejb" cluster (this is just an example. You might have to add username and password
# or callbackhandler depending on the security configuration(s) of these cluster node servers.
remote.cluster.ejb.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.cluster.ejb.connect.options.org.xnio.Options.SSL_ENABLED=false
# credentials if secured
#remote.cluster.ejb.username=...
#remote.cluster.ejb.password=...
Client application which is deployed in another EAP6.x instance
If an application uses the remote-outbound-connection, a jboss-ejb-client.xml configuration is needed. Here the cluster configuration needs to be added.
<jboss-ejb-client xmlns:xsi="urn:jboss:ejb-client:1.2" xsi:noNamespaceSchemaLocation="jboss-ejb-client_1_2.xsd">
<client-context>
<ejb-receivers>
<remoting-ejb-receiver outbound-connection-ref="remote-ejb-connection"/>
</ejb-receivers>
<!-- if an outbound connection connect to a cluster a list of members is provided after successful connection.
To connect to this node this cluster element must be defined.
-->
<clusters>
<cluster name="ejb" security-realm="ejb-security-realm" username="user">
<connection-creation-options>
<property name="org.xnio.Options.SSL_ENABLED" value="false" />
<property name="org.xnio.Options.SASL_POLICY_NOANONYMOUS" value="false" />
</connection-creation-options>
</cluster>
</clusters>
</client-context>
</jboss-ejb-client>
Related enhancement request
Root Cause
After the initial invocation with the remote.connections, the server which is finally invoked returns a cluster configuration with all available nodes in that cluster asynchronously.
The client tries to connect a subset of the provided nodes with a cluster configuration related to the returned cluster-name.
remote.clusters=ejb
These clusters must be defined in the client properties as the resolution describes. The default name of the standard configurations will be ejb.
Lets see how the cluster name is configured within the server configuration.
<subsystem xmlns="urn:jboss:domain:ejb3:1.3">
...
<stateful default-access-timeout="5000" cache-ref="simple" clustered-cache-ref="clustered"/>
...
<caches>
...
<cache name="clustered" passivation-store-ref="infinispan" aliases="StatefulTreeCache"/>
</caches>
<passivation-stores>
...
<cluster-passivation-store name="infinispan" cache-container="ejb"/>
</passivation-stores>
...
<subsystem xmlns="urn:jboss:domain:infinispan:1.3">
...
<cache-container name="ejb" aliases="sfsb sfsb-cache" default-cache="repl" module="org.jboss.as.clustering.ejb3.infinispan">
<transport lock-timeout="60000"/>
<replicated-cache name="repl" mode="ASYNC" batching="true">
<eviction strategy="LRU" max-entries="10000"/>
<file-store/>
</replicated-cache>
The configuration of ejb3.stateful points to the clustered cache which will be used for SFSB instances marked with @Clustered.
The cluster is defined as a cache of the EJB3 subsystem. This cache is backed by the passivation store, in this case named infinispan, and it is configured in the passivation-stores ,backed by a cache-container, which is in the infinispan subsystem in this example.
The cache-container="ejb" is the name of the cluster which you have to use in the client configuration.
Also the SLSB's, marked as @Clustered, uses the name. Only the name because there is no need for the cache/passivation.
Diagnostic Steps
Logging at client side
The defined connection can be used to call EJB's, all calls will return successful. But the following ERROR will be logged:
ERROR - JBREM000200: Remote connection failed: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed
INFO - Could not create a connection for cluster node ClusterNode{clusterName='ejb', nodeName='server-one', clientMappings=[ClientMapping{sourceNetworkAddress=/0:0:0:0:0:0:0:0, sourceNetworkMaskBits=0, destinationAddress='127.0.0.1', destinationPort=4447}], resolvedDestination=[Destination address=127.0.0.1, destination port=4447]} in cluster ejb
java.lang.RuntimeException: javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed
jboss-ejb-client.properties
Contains only
remote.connectionprovider.create.options.org.xnio.Options.SSL_ENABLED=false
remote.connections=default
remote.connection.default.host=localhost
remote.connection.default.port=4447
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOANONYMOUS=false
remote.connection.default.connect.options.org.xnio.Options.SASL_POLICY_NOPLAINTEXT=false
# some other entries start with
remote.connection*
Related Articles
Access EJB's if the EAP6 server is behind a LoadBalancer or NAT
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.