Remote EJB client hangs with high CPU after several hours and large number of invocations in EAP 7

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 7.2 CP8
    • 7.3 CP1

Issue

  • The server seem to stop processing EJB requests. When running a load test against a JBoss EAP 7.2 CP7 or JBoss EAP 7.3 CP1 server with stateful EJB calls from a Java client, after a some time, the server seem to stop responding to requests. On the client side there is no error, but on the client side is blocked with CPU at 100%. It does not matter if the client is a standalone Java client or if the Java client is located in another JBoss EAP server.

  • Migrated from EAP 6.4 to 7.2 and under production load system becomes unresponsive.

  • Thread dump shows:

      "default-threads - 47" #1718 prio=5 os_prio=0 tid=0x000000000f786800 nid=0x3f4e runnable [0x00007f4c9a094000]
      java.lang.Thread.State: RUNNABLE
          at org.jboss.remoting3._private.IntIndexHashMap.doGet(IntIndexHashMap.java:521)
          at org.jboss.remoting3._private.IntIndexHashMap.containsKey(IntIndexHashMap.java:178)
          at org.jboss.remoting3.util.InvocationTracker.addInvocation(InvocationTracker.java:122)
          at org.jboss.ejb.protocol.remote.EJBClientChannel.processInvocation(EJBClientChannel.java:332)
          at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.lambda$handleDone$0(RemoteEJBReceiver.java:93)
          at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1$$Lambda$1556/1096920767.notify(Unknown Source)
          at org.xnio.FinishedIoFuture.addNotifier(FinishedIoFuture.java:79)
          at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleDone(RemoteEJBReceiver.java:78)
          at org.jboss.ejb.protocol.remote.RemoteEJBReceiver$1.handleDone(RemoteEJBReceiver.java:76)
          at org.xnio.IoFuture$HandlingNotifier.notify(IoFuture.java:208)
          at org.xnio.AbstractIoFuture$NotifierRunnable.run(AbstractIoFuture.java:720)
          at org.xnio.IoUtils$2.execute(IoUtils.java:71)
          at org.xnio.AbstractIoFuture.runNotifier(AbstractIoFuture.java:693)
          at org.xnio.AbstractIoFuture$CompleteState.withNotifier(AbstractIoFuture.java:132)
          at org.xnio.AbstractIoFuture.addNotifier(AbstractIoFuture.java:570)
          at org.jboss.ejb.protocol.remote.RemoteEJBReceiver.processInvocation(RemoteEJBReceiver.java:134)

Resolution

  • Update to EAP 7.2.9+ or 7.3.2+

Root Cause

Diagnostic Steps

  • Get a series of thread dumps at the time of the issue.
  • Get a heap dump at the time of the issue and verify the size of the org.jboss.remoting3._private.IntIndexHashMap is very large.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.