Running out of memory with heavy EJB3 use on JBoss EAP

Environment

• JBoss Enterprise Application Platform (EAP)
  • 4.2
  • 4.3
  • 5.x

Issue

• Heap-based OutOfMemoryErrors (OOMEs) with heavy use of EJB3s due to EJB build up in ThreadLocalPools
• High CPU utilization with heavy use of EJB3s
• Heap dump shows org.jboss.ejb3.stateless.StatelessContainer taking up a significant percentage of the heap.

Resolution

Change the relevant EJB3 ThreadLocalPools to StrictMaxPools for your use case, and this will limit the instances of EJB3s that are created.
To do this, edit $JBOSS_HOME/server/$PROFILE/deploy/ejb3-interceptors-aop.xml and do a search and replace and change org.jboss.ejb3.ThreadlocalPool.class to org.jboss.ejb3.StrictMaxPool.class where it appears in the file.

For example, the Stateless Session Beans (SLSBs) domain in that file as specified looks like this:

  <domain name="Stateless Bean">
     <bind pointcut="execution(public * *->*(..))">
       <interceptor-ref name="org.jboss.ejb3.asynchronous.AsynchronousInterceptor"/>
       <interceptor-ref name="org.jboss.ejb3.ENCPropagationInterceptor"/>
       <interceptor-ref name="org.jboss.ejb3.security.AuthenticationInterceptorFactory"/>
     </bind>
     ...lines removed...
     <annotation expr="!class(@org.jboss.annotation.ejb.PoolClass)">
       @org.jboss.annotation.ejb.PoolClass (value=org.jboss.ejb3.ThreadlocalPool.class, maxSize=30, timeout=10000)
     </annotation>
  </domain>

and after the search and replace, it will look like this:

  <domain name="Stateless Bean">
     <bind pointcut="execution(public * *->*(..))">
       <interceptor-ref name="org.jboss.ejb3.asynchronous.AsynchronousInterceptor"/>
       <interceptor-ref name="org.jboss.ejb3.ENCPropagationInterceptor"/>
       <interceptor-ref name="org.jboss.ejb3.security.AuthenticationInterceptorFactory"/>
     </bind>
     ...lines removed...
     <annotation expr="!class(@org.jboss.annotation.ejb.PoolClass)">
       @org.jboss.annotation.ejb.PoolClass (value=org.jboss.ejb3.StrictMaxPool.class, maxSize=30, timeout=10000)
     </annotation>
  </domain>

Note: This search and replace will limit the changed EJB3 pools to 30 instances, so it may be necessary to adjust this amount depending on load by changing the maxSize attribute listed above.

Root Cause

• There are problems with the ThreadLocalPool implementation (backed by InfinitePool) in JBoss EAP 4.x. See Content from jira.jboss.org is not included.EJBTHREE-1175 and Content from jira.jboss.org is not included.EJBTHREE-1031 for a description of the situation. After these fixes, ThreadLocalPool is still an inherently an unbounded pool. It would stop growing when the threads using it stops growing, but if an unbounded amount of threads are used, then the ThreadLocalPool will grow indefinitely. ThreadLocalPool should not be used in these cases. The Timer Service is one example of an unbounded amount of threads that would cause ThreadLocalPool OOMEs.

Diagnostic Steps

• Obtain a heap dump (How do I create a Java heap dump?) when the OOME or high heap usage occurs and analyze it (How do I analyze a Java heap dump?).
• Determine if a large portion of the heap is being held by an EJB3 InfinitePool (org.jboss.ejb3.InfinitePool).

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.