How to tune the ejb3 subsystem in JBoss EAP 6

Solution Unverified - Updated 6 Aug 2024

Environment

Red Hat JBoss Enterprise Application Platform (EAP)
- 6.x

Issue

How to tune the ejb3 subsystem in JBoss EAP 6?

Recommendation setting for ejb subsystem in EAP 6

How to tune server to server EJB calls?
I can see this error repeatedly when JBoss is under stress:

javax.ejb.EJBException: JBAS014516: Failed to acquire a permit within 5 MINUTES

What is the recommended pool setting for ejb subsystem ?

Resolution

bean-instance-pools

These instance pools are used for Stateless Session EJBs and Message Driven Beans. For Stateful Session EJBs see [4] How to configure the Stateful EJB cache in JBoss EAP 6
These are instance pools not thread pools. The instance pool max-pool-size is the maximum number of EJB instances that can exist in the pool.
The default value is 20; this means that for a particular EJB there can only be 20 instances of that EJB at a time.
If there are more than 20 threads trying to use the particular EJB, then the client will block and wait for an instance to become available. The clients will wait for instance-acquisition-timeout and then throw an exception back if an instance has not become available. That exception looks like this:

Caused by: javax.ejb.EJBException: JBAS014516: Failed to acquire a permit within 10 SECONDS
	at org.jboss.as.ejb3.pool.strictmax.StrictMaxPool.get(StrictMaxPool.java:109) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
	at org.jboss.as.ejb3.component.pool.PooledInstanceInterceptor.processInvocation(PooledInstanceInterceptor.java:47) [jboss-as-ejb3-7.1.3.Final-redhat-4.jar:7.1.3.Final-redhat-4]
	at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.1.Final-redhat-2.jar:1.1.1.Final-redhat-2]
        ...

EJBs can use different pool configurations to allow finer control of the total number of each EJB type, however to find the suitable value for your environment, we would recommend testing.
The default EJB pool can be disabled and EJBs will not use a pool, instead a new EJB instance will be created when a given thread needs to invoke a method on an EJB, this is done by removing the bean-instance-pool-ref from the ejb3 subsystem as shown below:

         <subsystem xmlns="urn:jboss:domain:ejb3:1.3">
            <session-bean>
                <stateless>
                    <!-- remove this line -->
                    <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
                </stateless>
          ....

remote connector

1 thread for the WORKER_READ_THREADS & WORKER_WRITE_THREADS should typically be enough
A good default for MAX_INBOUND_MESSAGES & MAX_OUTBOUND_MESSAGES is to set it to the same size of your max-threads in the thread pool configuration.
If you want MAX_INBOUND_MESSAGES & MAX_OUTBOUND_MESSAGES to be larger than 80, the default value, you need to set them not only in the server side, but also in the client side as well.
MAX_INBOUND_MESSAGE_SIZE and MAX_OUTBOUND_MESSAGE_SIZE (in byte) are used to limit the message size to avoid unexpected large messages / memory consumption.

EJB Thread Pool

The EJB Thread Pools section defines the max-threads for the pool named default which by default is used for remote EJB requests, asynchronous EJB calls and EJB timers.
When using EJB Timers, Async EJBs, you may want to define a different thread pool for the EJB Timers and Async EJBs, this is especially useful if the EJB Timers or Async EJB calls are longer running processes.
The max-threads used by the <remote> are the maximum number of concurrent remote EJB requests that can be handled by the server. This is different from the instance pool configuration, which defines the maximum number of EJB instances that can be created for an EJB type. The EJB instances pool is used by remote EJB call threads as well as web http threads and other threads in the server.

         <subsystem xmlns="urn:jboss:domain:ejb3:1.3">
            <session-bean>
                <stateless>
                    <bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
                </stateless>
                <stateful default-access-timeout="5000" cache-ref="simple"/>
                <singleton default-access-timeout="5000"/>
            </session-bean>
            <mdb>
                <resource-adapter-ref resource-adapter-name="hornetq-ra"/>
                <bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
            </mdb>
            <pools>
                <bean-instance-pools>
                    <strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                    <strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
                </bean-instance-pools>
            </pools>
            <caches>
                <cache name="simple" aliases="NoPassivationCache"/>
                <cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>
            </caches>
            <passivation-stores>
                <file-passivation-store name="file"/>
            </passivation-stores>
            <async thread-pool-name="default"/>
            <timer-service thread-pool-name="default">
                <data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>
            </timer-service>
            <remote connector-ref="remoting-connector" thread-pool-name="default">
                <channel-creation-options>
                    <option name="WORKER_READ_THREADS" value="2" type="xnio"/>
                    <option name="WORKER_WRITE_THREADS" value="2" type="xnio"/>
                    <option name="MAX_INBOUND_MESSAGES" value="100" type="remoting"/>
                    <option name="MAX_OUTBOUND_MESSAGES" value="100" type="remoting"/>
                    <option name="MAX_INBOUND_MESSAGE_SIZE" value="100000" type="remoting"/>
                    <option name="MAX_OUTBOUND_MESSAGE_SIZE" value="100000" type="remoting"/>
                </channel-creation-options>
            </remote>
            <thread-pools>
                <thread-pool name="default">
                    <max-threads count="100"/>
                </thread-pool>
            </thread-pools>
            <iiop enable-by-default="false" use-qualified-name="false"/>
        </subsystem>

Exceptions that indicate tuning might be required:

Stateless EJB Instance pool not large enough or timeout too low

javax.ejb.EJBException: JBAS014516: Failed to acquire a permit within 20 SECONDS
     at org.jboss.as.ejb3.pool.strictmax.StrictMaxPool.get(StrictMaxPool.java:109)

EJB thread pool not large enough or EJB is taking longer to process than invocation timeout

See [3]. Note: If the EJB remoting thread-pool max-threads is too low, such that all threads are busy and a client tries to invoke an EJB, it will timeout with the invocation timeout if a thread does not become available.

java.util.concurrent.TimeoutException: No invocation response received in 300000 milliseconds

Trying to send more requests/responses over a channel and hit MAX_INBOUND_MESSAGES or MAX_OUTBOUND_MESSAGES limit. See [2].

org.jboss.remoting3.ChannelBusyException

Diagnostic Steps

Note: When tuning the size of the EJB pools, coming up with a calculated deterministic number of threads will usually not be feasible.

A better option is to design a stress test that emulates a real world scenario and run it with different sizes of thread pools, measuring request and over-whole performance for each configuration.

SBR

JBoss Base AS

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Performance tune

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.