How to configure and tune the session replication for EAP

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7
  • 6

Issue

• Is it possible to tune the session replication for a better performance
• What are the different valid options for the infinispan configuration for web session replication
• We use an environment where multiple requests reference the same sessionID, this cause performance issues and error messages within the logfile, is there a way to tune it?
• There are the following messages in the logfile, is there a configuration to avoid it?

ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-23) ISPN000136: Error executing command GetKeyValueCommand, writing keys []: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key

Resolution

Below are described some aspects for tuning, HTTP session stickyness, JGroups, Infinispan cache optimization, and finally Programmatically tuning. A related solutions section is at the end.

HTTP session stickyness

In general use session stickiness if a load balancer is used, otherwise this will have considerable performance impacts.
Without stickiness the session needs to be transferred to different nodes for each request, this might cause performance issues and locking contention.

JGroups cluster communication

The JGroups configuration which is used for the cluster communication is important for performance. The recommended stack is UDP because it uses multicast for the communication instead of unicast (TCP), this means for most requests a single message is sent for all nodes of the cluster instead of sending a separate message to each which burdens the instance and the network.
TCP must be used if there are network restrictions for UDP and multicast. The TCP configuration will not have a huge performance impact for clusters with a small number of nodes <=4

Infinispan subsystem "web" cache configuration

By default, the configuration will lock the session for each request to prevent concurrent read and modification.
The default cache of the web container will be used if not configured differently by specifying replication-config and cache-name in the jboss-web.xml.

Cache mode

• EAP 7.1 and further
  Consider the infinispan cache replication is marked as synchronous, but from a client perspective, the behavior is asynchronous even though from a server thread perspective the processing is synchronous because the Infinispan cache transaction is committed after the response is sent to the client.
  The cache mode should be always SYNC which is the default. Using ASYNC mode will cause locking contention and is not considered as valid and should not be used!
  Note from 7.2 onwards the attribute mode for the cache is deprecated and SYNC is used.
• EAP 7.0
  ASYNC is the default and can be used, but it is possible to use SYNC as well as it is potentially unnecessary to use a separate thread for cache commit
• EAP 6
  The cache mode is always ASYNC as the cache commit happens before the response is flushed.

As long as the transaction mode is BATCH this will cause locking because the Infinispan transaction will start with the request and commit when returning to the client.

Use the default configuration:

<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
   ...
   <distributed-cache name="dist">
      <locking isolation="REPEATABLE_READ"/>
      <transaction mode="BATCH"/>
      <file-store/>
   </distributed-cache>
</cache-container>

If the transaction mode is set to NONE every update will be a separate transaction. The downside of this is that the cache is replicated multiple times (on each commit) and reads to the session might get interim states of the session.
We recommend using ATTRIBUTE granularity to help in preventing performance issues.
If there is no control over these attribute classes (e.g. they come from a web framework), then the next best optimization is to use Transaction mode = BATCH,

Use the READ_COMMITTED/NONE if locking and transaction are removed, as below:

<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
   ...
   <distributed-cache name="dist">
      <locking isolation="READ_COMMITTED"/>
      <transaction mode="NONE"/>
      <file-store/>
   </distributed-cache>
</cache-container>

So locking READ_COMMITTED and Transaction mode NONE will allow concurrent access but with the drawback that it is possible to read stale attributes in concurrent sessions. But will have a performance penalty as well.

Valid configuration combinations:

There are two combinations of locking/transaction are the recommendation and no other combination should be used.

If you change to READ_COMMITTED/NONE to allow concurrent access, then it will no longer be able to block against concurrent access on different nodes. (However, that should not be happening with properly working sticky sessions). In EAP 7 the locking mechanism changed, EAP 6 had a separate component that locks the session across the cluster before it ever even called Infinispan.
This was removed in EAP 7 and it just used Infinispan locking instead, which no longer occurs when changing the config to allow concurrent access.

For example, the OPTIMISTIC lock approach is not supported for the web cache in EAP, and READ_COMMITTED/BATCH can cause issues as well.

Optimize the objects within the session programatically

First, we recommend keeping the amount of data stored in the session as small as possible. Storing a lot of data in the session objects can have an impact on the performance of the session replication. (For example, storing a large size of the object as the session attribute, storing many numbers of objects as the session attributes, or accessing mutable session attributes frequently.) Make sure to examine load testing results using your real-world application to determine a proper balance of data storage and performance to effectively use sessions.

Also, there are some options described below to optimize objects within the sessions, in a programmatic approach, i.e with code change:

Annotate immutable objects

A way to prevent from writing session attributes and trigger a replication for the cache is to mark the different attributes in the session as Immutable.
EAP assumes all session attributes are mutable, unless its of a known immutable type (JDK classes) or is annotated to be immutable (via @org.wildfly.clustering.web.Immutable or @net.jcip.Immutable). (See "Immutable Session Attributes" section in the development guide for details about a known immutable value/type.)
This means that a call to HttpSession.getAttribute(...) can still result in a write and trigger the replication.

If objects which are considered as immutable correctly annotated it will reduce the writes if a session will only read the attributes.

Change the granularity of session attributes within the cache

Using ATTRIBUTE granularity, only those attributes that were modified within the scope of a request are replicated.
This policy is not appropriate if an object reference is shared by multiple session attributes.
It can be more efficient than SESSION granularity if the session attributes are sufficiently large and/or modified infrequently.

The granularity can be changed from SESSION to ATTRIBUTE by using the following jboss-web.xml descriptor in your application.

<jboss-web version="8.0" xmlns="http://www.jboss.com/xml/ns/javaee"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://www.jboss.com/xml/ns/javaee http://www.jboss.org/j2ee/schema/jboss-web_8_0.xsd">
    <replication-config>
        <replication-granularity>ATTRIBUTE</replication-granularity>
    </replication-config>
</jboss-web>

Use different caches and configurations for different applications

If there is a need to have different session behavior the application can use a different cache configuration for session replication by adding the jboss-web.xml deployment descriptor. The cache-name tag can be added to the replication-config element and specify the cache using the name as <cache-container-name>.<cache-name>.

With the following configuration in the jboss-web.xml, the cache named myCache within the web cache-container is used:

    <replication-config>
        <cache-name>myCache</cache-name>
    </replication-config>

With the following configuration in the jboss-web.xml, the cache myCache from the cache-container myContainer is used:

    <replication-config>
        <cache-name>myContainer.myCache</cache-name>
    </replication-config>

Related Solutions

JBoss EAP 7 does not process requests for a session in parallel

Root Cause

To follow the Java EE specification for servlets, which says multiple nodes cannot concurrently access a session, the default configuration is most conservative to be spec compliant.

BATCH + PESSIMISTIC locking + REPEATABLE_READ

The configuration use BATCH + PESSIMISTIC locking + REPEATABLE_READ, which means sessions are locked per request, so concurrent access will effectively be serialized.
This will be slow but consistent when using sticky sessions, but is more or less unusable if not using sticky sessions.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.