Comparing EAP 7 invalidation cache vs hotrod-session-management implementations for externalization

Environment

• Red Hat Enterprise Application Platform
  • 7.x
• Red Hat Data Grid (RHDG)
  • 8.x
• Red hat OpenShift Container Platform (OCP)
  • 4.x

Issue

What are the differences between invalidation cache vs hotrod-session-management when setting cache externalization on EAP 7?

Resolution

Invalidation cache

This type of externalization was always present on EAP 7.

Example Implementation on cli

embed-server --std-out=echo --admin-only --server-config=standalone-openshift.xml 
/socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-rhdg-server:add(host=example-infinispan.dg-test.svc.cluster.local,port=11222)

batch
/subsystem=infinispan/remote-cache-container=rhdg:add(default-remote-cluster=data-grid-cluster,\
        properties={infinispan.client.hotrod.use_auth=false})
/subsystem=infinispan/remote-cache-container=rhdg/remote-cluster=data-grid-cluster:add(socket-bindings=[remote-rhdg-server])
run-batch
batch
/subsystem=infinispan/cache-container=web/invalidation-cache=infinispan:add()
/subsystem=infinispan/cache-container=web/invalidation-cache=infinispan/store=hotrod:add(remote-cache-container=rhdg,\
                fetch-state=false,\
                purge=false,\
                passivation=false,\
                shared=true)
/subsystem=infinispan/cache-container=web/invalidation-cache=infinispan/component=transaction:add(mode=BATCH)
/subsystem=infinispan/cache-container=web/invalidation-cache=infinispan/component=locking:add(isolation=REPEATABLE_READ)
/subsystem=infinispan/cache-container=web:write-attribute(name=default-cache,value=infinispan)
run-batch
stop-embedded-server

Result

The cli command above result in:

 <outbound-socket-binding name="remote-rhdg-server">
            <remote-destination host="example-infinispan.dg-test.svc.cluster.local" port="11222"/>
        </outbound-socket-binding>
…
<remote-cache-container name="rhdg" default-remote-cluster="data-grid-cluster">
                <property name="infinispan.client.hotrod.use_auth">false</property>
                <remote-clusters>
                    <remote-cluster name="data-grid-cluster" socket-bindings="remote-rhdg-server"/>
                </remote-clusters>
            </remote-cache-container>
…
                
<invalidation-cache name="infinispan">
                    <locking isolation="REPEATABLE_READ"/>
                    <transaction mode="BATCH"/>
                    <hotrod-store remote-cache-container="rhdg" fetch-state="false" passivation="false" purge="false" shared="true"/>
                </invalidation-cache>

hotrod-session-management

This type of externalization was implemented on EAP 7.3+ and has different expectations and behavior.
That was implemented on This content is not included.WFLY-7719.

Example CLI

embed-server --std-out=echo --admin-only --server-config=standalone-openshift.xml 
/socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-rhdg-server:add(host=example-infinispan.dg-test.svc.cluster.local,port=11222)
batch
/subsystem=infinispan/remote-cache-container=rhdg:add(default-remote-cluster=data-grid-cluster,\
        properties={infinispan.client.hotrod.use_auth=false})
/subsystem=infinispan/remote-cache-container=rhdg/remote-cluster=data-grid-cluster:add(socket-bindings=[remote-rhdg-server])
/subsystem=infinispan/remote-cache-container=rhdg:write-attribute(name=modules,    value=["org.wildfly.clustering.web.hotrod"])
run-batch
/subsystem=distributable-web/hotrod-session-management=mycache:add(granularity=ATTRIBUTE, remote-cache-container=rhdg)
/subsystem=distributable-web/hotrod-session-management=mycache/affinity=local:add()
run-batch
stop-embedded-server

Result of the cli command above

<outbound-socket-binding name="remote-rhdg-server">
            <remote-destination host="example-infinispan.dg-test.svc.cluster.local" port="11222"/>
        </outbound-socket-binding>
..
<remote-cache-container name="rhdg" default-remote-cluster="data-grid-cluster" modules="org.wildfly.clustering.web.hotrod">
                <property name="infinispan.client.hotrod.use_auth">false</property>
                <remote-clusters>
                    <remote-cluster name="data-grid-cluster" socket-bindings="remote-rhdg-server"/>
                </remote-clusters>
            </remote-cache-container>
…
 <hotrod-session-management name="mycache" remote-cache-container="rhdg" granularity="ATTRIBUTE">
                <local-affinity/>

On the example above host="example-infinispan.dg-test.svc.cluster.local" is the DG deployed in OCP 4.
And /subsystem=distributable-web/hotrod-session-management=mycache/affinity=local:add() is just for clarity, because the default behavior is already local.
And for non-clustering behavior - add the following two commands:

/subsystem=distributable-web/infinispan-session-management=default/affinity=local:add()
/subsystem=distributable-web/routing=local:add()

First will set the infinispan-session-management's affinity to local and the second will set the routing to local.

Comparative Difference invalidation cache vs hotrod-session-management

These are technically different session manager implementations, although conceptually they are similar, there are a few major differences on the implementation/usage.
When using the invalidation cache, sessions are always cached locally, and "passivation", i.e. exceeding max-active-sessions, involves evicting local entries. And the invalidation cache supports batching, whereas the hotrod-session-management does not. Whereas when using hotrod-session-management, local session caching is done via the near cache.
Major difference: The implementation of session expiration:

• when using infinispan-session-management (via an embedded cache, including invalidation), session expiration is handled within EAP, which schedules expiration tasks on the primary owner, reschedules on membership changes, etc.
• in the hotrod-session-manager, we have no primary ownership - thus expiration is delegated to RHDG

EAP uses client listeners to detect expiration, and uses a cumbersome first write wins operation to ensure session expiration is handled by a single cluster member the only other difference is in the preferred load balancing strategy.

Comparative Difference hotrod-session-manager vs embedded Infinispan session manager vs

With the embedded Infinispan session manager, the cache can be configured to be transactional with restrictive isolation that prevents concurrent access by multiple pods. However, EAP 7 cannot prevent this with the hotrod session manager (with its current implementation).

About EAP clustering requirements

Requirements for externalization with invalidation-cache:

• Sharing sessions requires all of the EAP instances to be clustered together;
• For servlet sessions, sticky sessions are always required regardless of use case or whether DG is used or not.
• Route (haproxy) in OCP 4.x has sticky session set by default with random strings for cookies. User can pass the one generated or create its own, when accessing EAP externalizing.

Requirements for externalization with hotrod-session-manager (and embedded session manager)

• LoadBalancer to have sticky sessions/upper layer to avoid only one session_id per pod, i.e session affinity
• All concurrent requests for a given session ID must be handled by the same pod, ie. sticky sessions

The requirements above are not unique to the hotrod session manager, same is required for the embedded session manager.

Root Cause

Invalidation cache vs hotrod-session-management are technically different session manager implementations and set differently on EAP 7 configuration.
This is explained on the solution Externalize EAP 7 HTTP Sessions To RHDG 8.

About jboss-web.xml

If the jboss-web.xml config contains a <replication-config/>, a equivalent configuration so, this can only ever create an infinispan-session-management therefore we need to get rid of that it is not ignored until EAP 8.
This will affect EAP 7, therefore.

FAQ

Q1. When using a Route to connect to EAP (deployed in OCP) can have an impact on performance on distributed and invalidation caches?
A1. Yes, the problem is that the Route object does not expose the requisite HAProxy configuration required such that it does not overwrite the response with its own cookie, but rather preserves any cookie that already exists.
Both distributed and invalidation caches can suffer (performance wise) from requests not getting directed to the primary owner. This manifests as a potentially remote call to schedule expiration, and in the case of distribution, the potential need to fetch state from the primary owner (unless an L1 is configured). The hotrod-based session manager will not have these issues.

Q2. The DG externalization, regardless of how is done, will have overhead. And the TPS will be lower than without externalization. Correct?
A2. Yes. Externalization will always have higher network utilization, due to the extra remote invocation to RHDG.
That's unavoidable.

Q3. In terms of comparing performance, would be worth testing hotrod-session-management vs invalidation caches?
A3. Yes, this is a worthwhile exercise. It would also be worth comparing performance during (and after) scaling operations.

Diagnostic Steps

1. To diagnose:  /subsystem=distributable-web:read-resource(recursive=true) - which will print the distributable-web

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.