Differences between Data Grid 8.3 vs Data Grid 8.2

Environment

• Red Hat Data Grid (RHDG)
  • 8.3 vs 8.2

Issue

Differences between RHDG 8.3 vs RHDG 8.2

Resolution

See the table below for DG 8.3 vs 8.2 differences, for differences with Dg Operator see here.

Gossip Router

DG Operator 8.3.x introduced the Xsite exchange of data via GossipRouter pod via tunnel protocol, which is unbounded so it should use the limit set on the limit range and the tunnel protocol doesn't have a timeout/heartbeat. Each site will have one, but JGroups only needs a single GossipRouter and if one of the sites crashes, it uses the other site's GossipRouter.
The following diagram shows Tunnel protocol usage:

custer1Node -> cluster1Master -> GossipRouter -> cluster2Master -> cluster2OtherNode
which is the same as:
cluster1Node -> cluster1Relay1 -> GossipRouter -> cluster1Relay2 -> cluster2OtherNode

Whereas without TUNNEL just remove the step in the middle,

custer1Node -> cluster1Master -> cluster2Master -> cluster2OtherNode

The Operator on DG 8.3.x sets the Tunnel protocol, which is not the default on the standalone version, by doing so it uses a Gossip Router. More information on the Gossip router see solution DG 8 operation in case of OCP nodes crashing

metrics and number of entries

DG 8.3 will display the approximate number of entries given statistics="true". To display the actual amount
(local+clustered+container NumberOfEntries+NumberOfEntriesInMemory) - use accurate-size="true" in the configuration. This avoids unnecessary calculations/access costs:

## infinispan.xml
cat infinispan.xml
   <cache-container name="default" statistics="true">
      <metrics accurate-size="true"/>
      <transport cluster="${infinispan.cluster.name:cluster}" stack="${infinispan.cluster.stack:tcp}" node-name="${infinispan.node.name:}"/>
   </cache-container>
## caches.xml
cat caches.xml
<infinispan xmlns="urn:infinispan:config:13.0">
    <cache-container>
        <caches>
            <invalidation-cache name="mycache" mode="ASYNC" statistics="true">
                <locking concurrency-level="1000" acquire-timeout="15000" striping="false"/>
            </invalidation-cache>

CPU throttling

The throttling should occur when the application's threads have used up its quota (limit), given the cpu quota the process is preempted. Depending on the kernel version this plays less of a role though.
OCP might use CFS’s quota mechanism to implement process limits, and therefore impose a limit on the threads as a consequence, regardless of cpu and memory settings.

JVM process/Infinispan process

JVM process is inelastic, meaning it will not reduce/fluctuates (as in cut back) - the value is set at start up. So the JVM process will start with the cpu limit as upper boundary. It will not take the lower boundary. So if the user sets limit 60 and request 6, it will use 60 as measure to calculate the cpus. If one sets limit == 1, so then the JVM impose the serial GC.
This is important for use cases with cpu throttling, where one sets a higher limit from the request to avoid cpu throttling.

Infinispan process will take limit (not request) as upper boundary

Because of the above explanation, i.e. jvm is inelastic, when using OCP, using Infinispan.spec.container.cpu limit:request, this will define the requested and the upper boundary of the usage. However the JVM (ie. the DG process) will use the whole limit for jvm and the internal processes, e.g. netty threads calculation:

apiVersion: infinispan.org/v1
kind: Infinispan
spec
  container:
    cpu: 60000m:6000m

The above behavior can be generalized for any Java product using Red Hat's build of OpenJDK images.

Details on non-blocking threads number of cpus and setting

DG 8.3.x has a merge of threads (merge non-blocking threads with netty threads) where the result is only non blocking, which makes the setting of worker-threads/core-threads attributes irrelevant - this applies to any DG installation (zip, Helm Charts, and Operator installation) regardless because this is related to the DG 8.3 server code itself.
The number of threads is calculated by number_of_cpus (defined on OCP via Infinispan.spec.container.cpu limit) but each thread will have a cache (since Netty sets it per thread). So for each thread, there will be a cache, which a limit is 16mb.

Heap usage as a consequence of the points above

Taking into consideration the above - namely:

• the JVM process will use Infinispan.spec.container.cpu limit as the upper boundary (for JVM cpu calculations)
• and the threads will be scaled linearly accordingly (netty == number_of_cpus*2)
• each thread will have a default cache usage (Netty 4.7 usage) up to 16mb each on heap

So in an Infinispan.spec.container.cpu: 60000m:6000m scenario, where the limit is 60 cores, there will be 60* 2 netty threads and other non-blocking threads, each with its netty buffers. Therefore 200+ non-blocking threads consume a total 200*16mb (size of the cache) = 2.3gb of heap.

To prevent the above scenario (high heap usage because of netty caching) set io.netty.allocator.useCacheForAllThreads to false consequently, it won't have a new cached pool for each thread and avoid such a high heap consumption.
Note that setting io.netty.allocator.type=unpooled is not recommended, and as it will have a performance hit as every new request will have to create a block of memory to handle the request.

More details can be found on Netty tuning for Data Grid 8.

Root Cause

See solution Using custom configuration in DG 8 via Operator: for custom usage of non-blocking threads.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.