Custom non-blocking threads setting on DG 8 Helm Charts

Solution Verified - Updated

Environment

  • Red hat OpenShift Container Platform (OCP)
    • 4.x
  • Red Hat Data Grid (RHDG)
    • 8.x
    • Helm Charts

Issue

How to set custom non-blocking threads settings on DG 8 Helm Charts?

Resolution

Use the following configuration to change the number of non-blocking threads on DG 8 deployed via Helm Charts:

infinispan.yml: |-
    infinispan:
      threads:
        threadPools:
          non-blocking-pool:
            non-blocking-bounded-queue-thread-pool:
              max-threads: 2
      cacheContainer:
        name: default
        statistics: true
        non-blocking-executor: non-blocking-pool
        security:
          authorization: {}

Root Cause

The solution Using custom configuration in DG 8 via Operator is about setting a custom configuration on Operator CR.

As explained in the solution Differences between DG 8.3 vs DG 8.2, there are some changes on DG 8.3 caused by the consolidation for non-blocking threads, which are calculated from the number of cpus available on the deployment (via cgroups):
Details on non blocking threads number of cpus and setting and the default value is 2* number of cpus.

DG 8.3.x has a merge of threads (merge non-blocking threads with netty threads) where the result is only non blocking, which makes setting of worker-threads/core-threads attributes irrelevant - this applies to any DG installation (zip, Helm Charts, and Operator installation) regardless because this is related to the DG 8.3 server code itself.
The number of threads is calculated by number_of_cpus (defined on OCP via Infinispan.spec.container.cpu limit) but each thread will have a cache (since Netty sets it per thread). So for each thread, there will be a cache, which limit is 16mb.

Use case

Some use cases with several memcached sharing resources, for instance, or a high number of cpu (set on OCP layer - which implicate a high number of non-blocking threads) so then the number of threads can be tuned (up or down accordingly).

Diagnostic Steps

Thread dump is the way to find the thread pool size and count the "non-blocking" threads:

2022-10-11 18:44:09,599 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.topology.LocalTopologyManagerImpl] Node data-grid-0-17555 joining cache memcached_node-224
2022-10-11 18:44:09,600 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.topology.ClusterCacheStatus] Queueing rebalance for cache memcached_node-224 with members [data-grid-0-17555]
2022-10-11 18:44:09,601 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.topology.LocalTopologyManagerImpl] Updating local topology for cache memcached_node-224: CacheTopology{id=1, phase=NO_REBALANCE, rebalanceId=1, currentCH=DefaultConsistentHash{ns=256, owners = (1)[data-grid-0-17555: 256+0]}, pendingCH=null, unionCH=null, actualMembers=[data-grid-0-17555], persistentUUIDs=[bb6374d8-170f-4697-82c0-88f1fb18d953]}
2022-10-11 18:44:09,601 DEBUG (non-blocking-thread--p2-t2) [org.infinispan.statetransfer.StateConsumerImpl] Removing no longer owned entries for cache memcached_node-224
2022-10-11 18:44:09,604 DEBUG (main) [org.infinispan.commons.util.ServiceFinder] Loading service impl: org.infinispan.persistence.remote.upgrade.HotRodTargetMigrator
2022-10-11 18:44:09,605 DEBUG (main) [org.infinispan.cache.impl.CacheImpl] Started cache memcached_node-224 on data-grid-0-17555

In attachment local_verification.sh (752 bytes) that can be used to count the number of threads.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.