How to use rolling upgrade for RHDG

Solution Verified - Updated

Environment

  • Red Hat Data Grid (RHDG)
    • 6
    • 7

Issue

  • Is it possible to use the rolling upgrade feature for migrate from JDG6 to JDG7?
  • Can I use rolling upgrade if I use embedded mode and not the JDG server?
  • When to use Rolling Upgrade procedure?
  • Using the Rolling Upgrade procedure described in JDG 7.0 or 7.1 documentation will cause issues, what is the reason?
  • Following the documentation for Rolling Upgrade - Administration Guide 7.1 and use --dumpkeys there are issues, what is wrong with it?
  • After performing a rolling upgrade operation, the DG target server when restarted still connects to the source server

Resolution

Rolling Upgrade is only available for Client/Server mode, for embedded mode it will not be possible.

Due to issues with older versions this feature is working with source clusters of the latest 6.6, 7.0 releases and 7.1 onwards, see Root Cause below.
The procedure (in a nutshell) is like this:

1)
Start a new cluster ( Target Cluster ) with the new version of RHDG, using different network settings for JGroups cluster so that the old cluster ( Source Cluster ) and the new one don’t overlap.
For each cache to be migrated, the Target Cluster is configured with a RemoteCacheStore with the following settings:

the remote-server config should point to the Source Cluster via the outbound-socket-binding property. See below the example.
remoteCacheName must coincide with the name of the cache on the Source Cluster
protocol-version must reflect the Hot Rod protocol version of the source cluster. For JDG 7.0.0, it's 2.5

<distributed-cache ...>
   <remote-store cache="MyExampleCache" socket-timeout="60000" tcp-no-delay="true" protocol-version="2.5" shared="true" hotrod-wrapping="true" purge="false" passivation="false">
      <remote-server outbound-socket-binding="remote-store-hotrod-server"/>
   </remote-store>
</distributed-cache>
...
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
  ...
  <outbound-socket-binding name="remote-store-hotrod-server">
     <remote-destination host="1.2.3.4" port="11222"/>
  </outbound-socket-binding>
  ...
</socket-binding-group>

2)
Configure all the clients to point to the Target Cluster instead of the Source Cluster , and one by one, restart each client node. Gradually, all requests will be handled by the Target Cluster rather than the Source Cluster . The Target Cluster will lazily load data from the Source Cluster on demand via the RemoteCacheStore.

3)
At this point the Target Cluster needs to fetch all remaining data from the Source Cluster.
Important to know: after starting this migration step updates from clients which are connected to the Source Cluster might be lost, this depends on whether the migration has been started for this segment
Step can be started via the CLI:

bin/cli.sh --connect controller=127.0.0.1:9990 -c "/subsystem=datagrid-infinispan/cache-container=clustered/distributed-cache=MyExampleCache:synchronize-data(migrator-name=hotrod)"

4)
Once the above operation is complete, the RemoteCacheStore on the Target Cluster must be disabled. This can be done via CLI:

bin/cli.sh --connect controller=127.0.0.1:9990 -c "/subsystem=datagrid-infinispan/cache-container=clustered/distributed-cache=MyExampleCache:disconnect-source(migrator-name=hotrod)"

This is to prevent the target instances to unnecessary connect the source cluster.

5)
The Source Cluster can be decomissioned now.

JDG 7 - Administration and Configuration Guide Rolling Upgrades
Content from infinispan.org is not included.Infinispan 9.3 - User Guide

Note that legacy documentation show the wrong procedure and steps!

Root Cause

Unfortunately the feature is broken in older releases and it is not possible to use Rolling Upgrade in any circumstances correctly, there is an Content from issues.jboss.org is not included.umbrella tracker.
The documentation until JDG 7.1 describe the legacy procedure which is not working correctly, the documentation of 7.2 will be updated and 7.3 or the upstream online documents are correct.

  • fixed for source and target cluster with version 6.6.2, 7.0.1+ and 7.1.0 onwards.
  • because of the issues there is a patch needed for older relases and there are no plans for this, it need to be checked individually. Please don't hesitate to open a case if you try to migrate and have questions or concerns.
  • legacy Documentation mentioned a step to dump the keyset for the source cluster, this can be skipped. Please follow the documentation for Rolling Upgrade in 7.2 or Content from infinispan.org is not included.upstream project documentation
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.