Externalize EAP 7 HTTP Sessions To RHDG 8
Environment
- Red Hat JBoss Enterprise Application Platform (EAP)
- 7.x
- Red Hat Data Grid (RHDG)
- 8.x
Issue
- How do I externalize HTTP sessions from EAP 7.x to RHDG 8?
Resolution
The steps below describe the process for externalization of http sessions from EAP 7's Application to DG 8. The steps are namely: In DG: create user and declare cache (for remote-store), and in EAP set invalidation cache (with cache store) and socket binding.
Create a user on RHDG:
$ RHDG/bin/cli.sh user create rhdgadmin -p password@2 -g admin
Add new RHDG cache to the existing RHDG default cache-container:
<cache-container name="default" statistics="true">
<transport cluster="${infinispan.cluster.name:cluster}" stack="${infinispan.cluster.stack:tcp}" node-name="${infinispan.node.name:}"/>
...
<distributed-cache name="myTestCache" owners="2"/>
</cache-container>
The cache can either be declared on the infinispan.xml or the caches.xml (located in $RHDG/server/data). To do it via cli command do:
create cache --template=org.infinispan.DIST_SYNC myTestCache <-- create cache myTestCache in caches.xml; owner=2 by default
Note: this step is required for remote-store, for hotrod store, the cache is automatically created (even if it is not declared).
On EAP, add an additional cache to EAP's existing web cache-container:
<cache-container name="web" default-cache="dist" module="org.wildfly.clustering.web.infinispan">
...
<invalidation-cache name="rhdg">
<locking isolation="REPEATABLE_READ"/>
<transaction mode="BATCH"/>
<remote-store cache="myTestCache" socket-timeout="60000" remote-servers="remote-rhdg-server" passivation="false" purge="false" shared="true"> <!-- remote-servers must match the socket binding below -->
<property name="infinispan.client.hotrod.auth_username">rhdgadmin</property> <!-- must match the users above -->
<property name="infinispan.client.hotrod.auth_password">password@2</property> <!-- must match the password above -->
<property name="infinispan.client.hotrod.auth_realm">default</property>
<property name="infinispan.client.hotrod.sasl_mechanism">SCRAM-SHA-512</property>
</remote-store>
</invalidation-cache>
</cache-container>
Above example is for invalidation type configuration - not hotrod-session-management, for more details on the difference see solution Comparing EAP 7 invalidation cache vs hotrod-session-management implementations for externalization.
Cli Commands
For CLI command settings, see solution JBoss EAP 7's infinispan CLI command to insert remote store cache.
Note: HTTP sessions will be stored in cache rhdg on EAP. Those sessions will be externalized to myTestCache created on RHDG. The hotrod client properties are used to allow EAP to connect to the remote RHDG instance. Depending on the configuration of the remote RHDG instance, additional properties may be required.
Add the outbound-socket-binding, pointing the RHDG instance:
<outbound-socket-binding name="remote-rhdg-server"> <!-- binding name must match the socket binding above -->
<remote-destination host="127.0.0.1" port="11222"/>
</outbound-socket-binding>
Example command to add the outbound-socket-binding:
/socket-binding-group=standard-sockets/remote-destination-outbound-socket-binding=remote-rhdg-server:add(host=127.0.0.1,port=11222)
Note that socket binding remote-rhdg-server correlates to the remote-servers defined in the remote-store configuration of the rhdg cache previously defined.
Within the deployed application's jboss-web.xml, specify the name of the EAP cache that will hold the externalized sessions:
<?xml version="1.0" encoding="UTF-8"?>
<jboss-web>
<context-root>/counter</context-root>
<replication-config>
<replication-trigger>SET_AND_GET</replication-trigger>
<replication-granularity>SESSION</replication-granularity>
<cache-name>web.rhdg</cache-name> <== ADDED
</replication-config>
</jboss-web>
Note that "web." is prefixed to the EAP cache name.
Start the RHDG instance, followed by the EAP instance. Note that EAP will automatically connect to the RHDG instance and begin to cache HTTP sessions to it once the deployed application is accessed. One can use the RHDG 8 console to view cache entries, metrics, configuration, etc. On the console access:
Content from 127.0.0.1 is not included.http://127.0.0.1:11222/console/cache/myTestCache
Console Metrics view:
Root Cause
Requirements for externalization
Requirements for externalization with invalidation-cache:
- Sharing sessions requires all of the EAP instances to be clustered together;
- For servlet sessions, sticky sessions are always required regardless of use case or whether DG is used or not.
- Route (haproxy) in OCP 4.x has sticky session set by default with random strings for cookies. User can pass the one generated or create its own, when accessing EAP externalizing.
Requirements for externalization with hotrod-session-manager (and embedded session manager)
- LoadBalancer to have sticky sessions/upper layer to avoid only one session_id per pod, i.e session affinity
- All concurrent requests for a given session ID must be handled by the same pod, ie. sticky sessions
Supportability
Only invalidated-cache should be used in EAP 7 (as above) when offloading sessions to RHDG. Any other modes are not supported and do not make sense, because they will waste resources and provide worse performance for that use case.
For replicated or distributed cache, every write will use extra bandwidth copying the data to other EAP nodes, and use X times more heap memory across the EAP cluster. Without providing any benefits, because the backup copies are in RHDG. Moreover, distributed-cache with one owner will be worse than invalidated, in cases where the primary cache owner is different than the node processing the requests.
DG 8 Protobuf requirement
DG 8 can only show entry as JSON if they are internally stored as ProtoBuf, however EAP 7 sessions are marshalled with JBoss Marshalling and even though it can be used, DG wouldn't have the protobuf definitions used by EAP
thus it cannot read them (even though it also uses protostream).
In this matter, session are not marshalled with JBoss Marshalled - session attributes are marshalled with the marshaller with which the session manager was configured (where the default is JBoss Marshalling)
The only way in which the DG server can transform the data into a readable JSON would be to deploy all of the classes that have been marshalled into the session on the DG side the only possibility would be for EAP to use ProtoBuf, which is not currently possible.
By default DG stores keys and values as byte[], and as long as the user doesn't request command from DG 8 to perform internal operations, that is absolutely fine.
Given EAP 7 implements Java Servlet specification, it allows for session attributes to be of type Object they can be anything the user application decides to put in there, not just Protobuf.
Currently EAP 7 doesn't support protostream marshaling, however EAP 8 will have this feature.
DG requires to understand the data in the following cases:
- query
- trancoding
- listener filtering
- schema-aware cache stores (SQLStore)
- server-side tasks
EAP to DG externalization in OCP 4
- For invalidation cache:
For invalidation sharing sessions requires the pod/nodes all to be clustered together, and this is not specific to OCP, that applies everywhere.
Consequently, sharing sessions between pods in different OCP deployments requires them all to be clustered together, this is usually done via DNS_PING or KUBE_PING depending on the case (for instance trying to connect two EAP deployment as cluster in different OCP clusters).
For servlet sessions, sticky sessions are always required regardless of use case or whether DG is used or not.
For EAP clustering in OCP see the solution: EAP 7 image clustering in OCP 4.
For DG clustering in OCP see the solution: Data Grid 8 Operator Clustering details.
By default OCP 4.x Route brings sticky sessions by default and a random string cookie. Again, the Route should bring sticky sessions by default.
- For hotrod-session-management
For externalization with hotrod-session-management, technically EAP is outsourcing its cluster to RHDG
so the EAP clustering is not a requirement but session affinity and sticky sessions become essential to guarantee that all concurrent requests for a given session ID should be handled by the same pod and that prevent any collision.
In this matter, only one pod can successfully create a session for a specific session_id
the others would return the session for the existing_id and all current requests for a given session ID must be handled by the same pod. That is why load balancers are meant to be configured with session affinity for example.
Example
Attached on this solution there are DG 8.3's infinispan/caches.xml files, and a deployment for testing, just rename if for verify-cluster.war instead of .txt.
Diagnostic Steps
By default OCP 4.x Route brings sticky sessions by default and a random string cookie. Meaning sessions are sticky by default unless disabled via route annotation, source lb strategy uses the source IP address instead of cookie.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.