EJB remote invocations are affected sometimes for a short period of time after an Exception is logged

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.x
    • 7.x

Issue

  • Sporadical a couple of EJBCLIENT000025 errors within the logfile, after enable DEBUG we saw the following DEBUG message
DEBUG [thread-3] org.jboss.ejb.client: Retrying invocation org.jboss.ejb.client.EJBClientInvocationContext@4b66500b which failed on node: Node1 due to:
org.jboss.ejb.client.RequestSendFailedException
	at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocationInternal(RemotingConnectionEJBReceiver.java:257)
	at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocation(RemotingConnectionEJBReceiver.java:215)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:184)
	at org.jboss.ejb.client.EJBObjectInterceptor.handleInvocation(EJBObjectInterceptor.java:58)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
	at org.jboss.ejb.client.EJBHomeInterceptor.handleInvocation(EJBHomeInterceptor.java:83)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
	at org.jboss.ejb.client.TransactionInterceptor.handleInvocation(TransactionInterceptor.java:42)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
	at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:125)
	at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:186)
	at org.jboss.ejb.client.EJBInvocationHandler.sendRequestWithPossibleRetries(EJBInvocationHandler.java:255)
	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:200)
	at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
	at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
	at com.sun.proxy.$Proxy0.hasConnection(Unknown Source)
	...
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1301)
	at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
	at org.jboss.ejb.client.remoting.ChannelAssociation.acquireChannelMessageOutputStream(ChannelAssociation.java:261)
	at org.jboss.ejb.client.remoting.RemotingConnectionEJBReceiver.processInvocationInternal(RemotingConnectionEJBReceiver.java:239)
	... 29 more
  • logfile shows messages that a server is excluded from selection for the application
DEBUG [thread-3] org.jboss.ejb.client.EJBClientContext: Node1 is excluded from handling appname=MyApp,modulename=ejb,distinctname= in invocation context org.jboss.ejb.client.EJBClientInvocationContext@2a55600b

Resolution

The issue is tracked by
This content is not included.EAP6 BZ1360391
Content from issues.jboss.org is not included.EAP7.0 JBEAP-5411
Content from issues.jboss.org is not included.EAP 7.1 JBEAP-5413

The bug was fixed for EAP 6.4.19 and EAP 7.1 so EAP 6.4 users should apply CP19 and EAP 7 users should upgrade to EAP 7.1 and apply the latest CP.

Root Cause

If EJB invocations can't be completed the ejb-client mark the channel/connection as broken and therefore a hickup is seen in the logfiles.
The stack trace shows an InterruptedException which mean the client processed the remote invocation in a different thread and canceled it for some reason. This can be the case if swing clients are used and cancel the thread during the invocation.
The issue is exactly that EJB decides that if it can't read the return message, the channel must be broken, and if the channel is broken, then the connection must be broken.
Really it's a wrong assumption; it should just report the error and carry on because the error is coming from jboss-marshalling, not the channel really.

Diagnostic Steps

  • Enable DEBUG at client side for org.jboss.ejb.client to check whether there are invocation failures, retry or excluded messages
  • check whether there are errors
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.