Remote EJB client hang when load increases in EAP 7

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7
• Enterprise Java Beans (EJB)
• JBoss EAP as EJB server.
• JBoss EAP as EJB client.
• protocol "remote+http(s)" or "remote"

Issue

• Remote EJB client hang in awaitResponse when load increases

    "default task-127" #285 prio=5 os_prio=0 tid=0x0000000028c9d800 nid=0x3a4c in Object.wait() [0x00000000425ae000]
     java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:502)
    at org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:413)
    - locked <0x00000006ccf82360> (a java.lang.Object)
    at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:204)
    at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:183)
    at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:146)
  

• Setting "invocation-timeout" in wildfly-config.xml or "invocation.timeout" in the InitialContext does not work.

Resolution

Apply EAP 7.2 CP2+ , EAP 7.3 is not affected.

Workaround:

Set the task-max-threads on the io subystem's default worker, and then add a request-limit filter and limit it to 10 less than the task-max-threads

/subsystem=io/worker=default:write-attribute(name=task-max-threads, value=200)
/subsystem=undertow/configuration=filter/request-limit=maxConcurrentRequests:add(max-concurrent-requests=190, queue-size=100)
/subsystem=undertow/server=default-server/host=default-host/filter-ref=maxConcurrentRequests:add()

        <subsystem xmlns="urn:jboss:domain:io:3.0">
            <worker name="default" task-max-threads="200"/>
            <buffer-pool name="default"/>
        </subsystem>
...
        <subsystem xmlns="urn:jboss:domain:undertow:7.0" default-server="default-server" default-virtual-host="default-host" default-servlet-container="default" default-security-domain="other">
               ...
                <host name="default-host" alias="localhost">
                    <location name="/" handler="welcome-content"/>
                    <filter-ref name="maxConcurrentRequests"/>
                    <http-invoker security-realm="ApplicationRealm"/>
                </host>
            </server>
           ....
            <filters>
                <request-limit name="maxConcurrentRequests" max-concurrent-requests="190" queue-size="100"/>
            </filters>
        </subsystem>

Note: queue-size is required to be set in EAP 7.0 as it defaults to undefined, in EAP 7.1 it defaults to 0 so is not required to be specifically set.

Root Cause

Known issue This content is not included.JBEAP-14524 EJB race condition can cause client to be in awaitResponse while server is done or This content is not included.JBEAP-15737 for JBoss EAP 7.2.

Note:
The same issue seems to affect using the older remote protocol (protocol="remote"), but there is no known workaround other than to update to "remote+http"/"remote+https".

Diagnostic Steps

• Generate Java thread dumps of both the client and server.

  • The client will have many threads with
    org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse(EJBClientInvocationContext.java:413)

  • The server should not show any of the EJB calls still running

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.