Using custom configuration in DG 8 via Operator

Solution Verified - Updated

Environment

  • Red hat OpenShift Container Platform (OCP)
    • 4.x
  • Red Hat Data Grid (RHDG)
    • 8.x
    • DG Operator

Issue

How to use custom configuration in DG 8 via Operator?
Via DG 8 Operator, how to set custom configuration?

Resolution

User can use a ConfigMap to set custom configurations in DG Operator starting with DG Operator 8.2.x, via spec.configMapName - see example below.

Requirements

Setting the configuration via config-map is very flexible, but at the same time, the configuration must be valid, which is the same as the standalone configuration.
Meaning the stanzas (tags) must be properly set and organized, and the parameters must fit the schema files.

Limitations

The configuration is bounded in two aspects:

  1. The configuration should be accepted by the operator as valid, meaning creation of endpoints for instance must be allowed by the operator, otherwise the changes will be overwritten.
  2. The schema files organization

Location:

Inside the pod the configuration used by the operator is: /opt/infinispan/server/conf/operator/infinispan.xml (which also has a log4j.xml). And the user config will be in /opt/infinispan/server/conf/user/infinispan-config.yaml. And the caches created will be present on /opt/infinispan/server/data/caches.xml.

Cache filePurpose
/opt/infinispan/server/conf/operator/infinispan.xmlConfiguration used by the operator
/opt/infinispan/server/conf/user/date/infinispan-config.yamlConfiguration set by the user
/opt/infinispan/server/data/caches.xmlCaches declaration location

Process underneath

The Operator overlays its generated configuration over the xml provided by the user. So the sum of the two configurations need to result in a valid configuration that the server can parse.
Basically a combination of the user and the default configuration file, which is separated from the infinispan.xml in the image.

Examples:

For creating two caches:

    infinispan-config.xml: >
      <infinispan
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:infinispan:config:13.0 https://infinispan.org/schemas/infinispan-config-13.0.xsd
                                urn:infinispan:server:13.0 https://infinispan.org/schemas/infinispan-server-13.0.xsd"
            xmlns="urn:infinispan:config:13.0"
            xmlns:server="urn:infinispan:server:13.0">      <cache-container name="default"
                         statistics="true">
          <distributed-cache name="mycacheone"
                             mode="ASYNC"
                             statistics="true">
            <encoding media-type="application/x-protostream"/>
            <expiration lifespan="300000"/>
            <memory max-size="400MB"
                    when-full="REMOVE"/>
          </distributed-cache>
          <distributed-cache name="mycachetwo"
                             mode="SYNC"
                             statistics="true">
            <encoding media-type="application/x-protostream"/>
            <expiration lifespan="300000"/>
            <memory max-size="400MB"
                    when-full="REMOVE"/>
          </distributed-cache>
          <distributed-cache name="mycachethree"
                             mode="SYNC"
                             statistics="true">
            <encoding media-type="application/x-protostream"/>
            <expiration lifespan="300000"/>
            <memory max-size="400MB"
                    when-full="REMOVE"/>
          </distributed-cache>
        </cache-container>
      </infinispan>

For creating memcached endpoints:

  data:
    infinispan-config.xml: >
      <infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:13.0 https://infinispan.org/schemas/infinispan-config-13.0.xsd
                              urn:infinispan:server:14.0 https://infinispan.org/schemas/infinispan-server-13.0.xsd"
        xmlns="urn:infinispan:config:13.0"
        xmlns:server="urn:infinispan:server:13.0">

        <server  xmlns="urn:infinispan:server:13.0">
          <socket-bindings default-interface="public" port-offset="${infinispan.socket.binding.port-offset:0}">
             <socket-binding name="memcached" port="11221"/>
          </socket-bindings>
          <endpoints>
             <endpoint socket-binding="default" security-realm="default">
                   <rest-connector/>
                   <memcached-connector socket-binding="memcached"/>
                </endpoint>
          </endpoints>
          </server>
      </infinispan>

^ Above is not supported on DG Operator; Meaning memcached is not supported on DG Operator and requires the operator to be disabled to work - therefore not supported.

For setting accurate metrics

Example below is accurate metrics:

- apiVersion: v1
  kind: ConfigMap
  metadata: 
    name: "${CLUSTER_NAME}-custom-config"
    namespace: ${CLUSTER_NAMESPACE}
  data: 
    infinispan-config.xml: >
      <infinispan
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:infinispan:config:14.0 https://infinispan.org/schemas/infinispan-config-14.0.xsd
                                urn:infinispan:server:14.0 https://infinispan.org/schemas/infinispan-server-14.0.xsd"
            xmlns="urn:infinispan:config:14.0"
            xmlns:server="urn:infinispan:server:14.0">
             <cache-container name="default" statistics="true" non-blocking-executor="non-blocking-pool">
                <metrics accurate-size="true"/>
             </cache-container>
      </infinispan>

For setting the number of threads via config-map

Example below is to set the number of non-blocking-pool for 20 as an example:

- apiVersion: v1
  kind: ConfigMap
  metadata: 
    name: "${CLUSTER_NAME}-custom-config"
    namespace: ${CLUSTER_NAMESPACE}
  data: 
    infinispan-config.xml: >
      <infinispan
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:infinispan:config:14.0 https://infinispan.org/schemas/infinispan-config-14.0.xsd
                                urn:infinispan:server:14.0 https://infinispan.org/schemas/infinispan-server-14.0.xsd"
            xmlns="urn:infinispan:config:14.0"
            xmlns:server="urn:infinispan:server:14.0">
            <threads>
             <non-blocking-bounded-queue-thread-pool name="non-blocking-pool" max-threads="20"/> 
            </threads>
             <cache-container name="default" statistics="true" non-blocking-executor="non-blocking-pool">
                <metrics accurate-size="true"/>
             </cache-container>
      </infinispan>

And for non-blocking on Helm see Custom non-blocking threads setting on DG 8 Helm Charts.

Observation: core-threads attribute has no effect, so it can be removed above.

Setting server.logs on DG 8.3.x

Here is how to set server.log via config-map:

 - apiVersion: v1
  kind: ConfigMap
  metadata:
    name: "${CLUSTER_NAME}-custom-config"
    namespace: ${CLUSTER_NAMESPACE}
  data:
    log4.xml: >
<?xml version="1.0" encoding="UTF-8"?>
<Configuration name="InfinispanServerConfig" monitorInterval="60" shutdownHook="disable">
  <Appenders>
    <!-- Colored output on the console -->
    <Console name="STDOUT">
      <PatternLayout pattern="%highlight{%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %m%throwable}{INFO=normal, DEBUG=normal, TRACE=normal}%n"/>
    </Console>
    <!-- Rolling file -->
    <RollingFile name="FILE" createOnDemand="true"
                 fileName="${path}/server.log"
                 filePattern="${path}/server.log.%d{yyyy-MM-dd}-%i">
      <Policies>
        <OnStartupTriggeringPolicy />
        <SizeBasedTriggeringPolicy size="100 MB" />
        <TimeBasedTriggeringPolicy />
      </Policies>
      <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %m%throwable%n"/>
    </RollingFile>
  </Appenders>
  <Loggers>
    <Root level="INFO">
      <AppenderRef ref="STDOUT" level="INFO"/>
      <AppenderRef ref="FILE"/>
    </Root>
  </Loggers>
</Configuration>

Setting server.logs and access logson DG 8.3.x

Given the right appender and logger as below:

log4j.xml: >
      <?xml version="1.0" encoding="UTF-8"?>
      <Configuration name="InfinispanServerConfig" monitorInterval="60" shutdownHook="disable">
        <Appenders>
          <!-- Colored output on the console -->
          <Console name="STDOUT">
            <PatternLayout pattern="%highlight{%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %m%throwable}{INFO=normal, DEBUG=normal, TRACE=normal}%n"/>
          </Console>          <!-- Rolling file -->
          <RollingFile name="FILE" createOnDemand="true"
                       fileName="/opt/infinispan/server/log/server.log"
                       filePattern="/opt/infinispan/server/log/server.log.%d{yyyy-MM-dd}-%i">
            <Policies>
              <OnStartupTriggeringPolicy />
              <SizeBasedTriggeringPolicy size="100 MB" />
              <TimeBasedTriggeringPolicy />
            </Policies>
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %m%throwable%n"/>
          </RollingFile>
          <RollingFile name="HR-ACCESS-FILE" createOnDemand="true"
                       fileName="/opt/infinispan/server/log/hotrod-access.log"
                       filePattern="/opt/infinispan/server/log/hotrod-access.log.%d{yyyy-MM-dd}-%i">
            <Policies>
              <OnStartupTriggeringPolicy />
              <SizeBasedTriggeringPolicy size="100 MB" />
              <TimeBasedTriggeringPolicy />
            </Policies>
            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %m%throwable%n"/>
          </RollingFile>
        </Appenders>        <Loggers>
          <Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="TRACE">
             <AppenderRef ref="HR-ACCESS-FILE"/>
          </Logger>
          <Root level="INFO">
            <AppenderRef ref="STDOUT" level="INFO"/>
            <AppenderRef ref="FILE"/>
            <!-- if you set this, it will be a copy of server.log -->                     <!-- <AppenderRef ref="HR-ACCESS-FILE"/>-->
          </Root>
        </Loggers>
      </Configuration>

For details on the pattern to use on the access logs, see solution Enabling hotrod and REST access logs in Red Hat Datagrid 8.x - there is a complete discussion on the default pattern and the recommended pattern for access logs, which can be used to help on hot rod client investigation.

Setting security section

Set the security inside server and infinispan sections:

data:
    infinispan-config.xml: >
      <infinispan
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:infinispan:config:13.0 https://infinispan.org/schemas/infinispan-config-13.0.xsd
                                  urn:infinispan:server:14.0 https://infinispan.org/schemas/infinispan-server-13.0.xsd"
            xmlns="urn:infinispan:config:13.0"
            xmlns:server="urn:infinispan:server:13.0">

         <server  xmlns="urn:infinispan:server:13.0">
            <security>
               <credential-stores>
                  <credential-store name="credentials" path="credentials.pfx">
                     <clear-text-credential clear-text="secret"/>
                  </credential-store>
               </credential-stores>
               <security-realms>
                  <security-realm name="default">
                     <!-- Uncomment to enable TLS on the realm -->
                     <!-- server-identities>
                        <ssl>
                           <keystore path="application.keystore"
                                     password="password" alias="server"
                                     generate-self-signed-certificate-host="localhost"/>
                          <engine enabled-protocols="TLSv1.3" enabled-ciphersuites="TLS_AES_256_GCM_SHA384 TLS_AES_128_GCM_SHA256 TLS_AES_128_CCM_8_SHA256"/>
                        </ssl>
                     </server-identities-->
                     <properties-realm groups-attribute="Roles">
                        <user-properties path="users.properties"/>
                        <group-properties path="groups.properties"/>
                     </properties-realm>
                  </security-realm>
               </security-realms>
            </security>
            </server>
      </infinispan>

TLS 1.2 on DG Operator

data:
    infinispan-config.xml: |
      <infinispan
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:infinispan:config:13.0 https://infinispan.org/schemas/infinispan-config-13.0.xsd
                                  urn:infinispan:server:14.0 https://infinispan.org/schemas/infinispan-server-13.0.xsd"
            xmlns="urn:infinispan:config:13.0"
            xmlns:server="urn:infinispan:server:13.0">
         <server  xmlns="urn:infinispan:server:13.0">
            <security>
               <security-realms>
                  <security-realm name="default">
                     <server-identities>
                        <ssl>
                           <engine enabled-protocols="TLSv1.2" enabled-ciphersuites="TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA256"/> <!-- TLS 1.2 -->
                           <keystore path="/etc/security/conf/operator-security/keystore.pem" keystore-password=""/> <!-- attention here -->
                        </ssl>
                     </server-identities>
                     <properties-realm groups-attribute="Roles">
                        <user-properties path="cli-users.properties" relative-to="infinispan.server.config.path"/>
                        <group-properties path="cli-groups.properties" relative-to="infinispan.server.config.path"/>
                     </properties-realm>
                  </security-realm>
               </security-realms>
            </security>
            </server>
      </infinispan>

Note that because of jira This content is not included.JDG-6278, the section keystore path must be added. Starting on DG 8.4.4 (server, not operator) this won't be required anymore.

Setting a custom DNS_PING service

Create a custom stack and assign to the transport stack:

infinispan-config.xml: >
      <infinispan
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="urn:infinispan:config:13.0 https://infinispan.org/schemas/infinispan-config-13.0.xsd
                                urn:infinispan:server:13.0 https://infinispan.org/schemas/infinispan-server-13.0.xsd"
            xmlns="urn:infinispan:config:13.0"
            xmlns:server="urn:infinispan:server:13.0">
            <jgroups transport="org.infinispan.remoting.transport.jgroups.JGroupsTransport">
          <stacks>
            <stack name="custom-stack" extends="tcp">
                  <TCP port-range="0" bind-addr="SITE_LOCAL" bind-port="7800" enable-diagnostics="false"/>
                  <dns.DNS_PING dns-record-type="A" stack.combine="REPLACE" stack.position="MPING" dns-query="dg-cluster-test-ping-2.dg-test.svc.cluster.local"/>
              </stack>
          </stacks>
      </jgroups>
      <cache-container name="default" statistics="true" non-blocking-executor="non-blocking-pool" shutdown-hook="DONT_REGISTER">
        <transport cluster="dg-cluster-test" node-name="" stack="custom-stack"/>
      </cache-container>
      </infinispan>

Setting RESP protocol endpoint

apiVersion: v1
kind: ConfigMap
metadata: 
  name: infinispan-configuration-custom
  namespace: example
data: 
  infinispan-config.xml: |
    <infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:15.0 http://www.infinispan.org/schemas/infinispan-config-15.0.xsd
              urn:infinispan:server:15.0 http://www.infinispan.org/schemas/infinispan-server-15.0.xsd"
    xmlns="urn:infinispan:config:15.0"
    xmlns:server="urn:infinispan:server:15.0">
      <cache-container name="default">
        <distributed-cache name="example-memcached" />
      </cache-container>
      <server xmlns="urn:infinispan:server:15.0">
          <endpoints>
            <endpoint socket-binding="default" security-realm="default">
              <memcached-connector cache="example-memcached"/>
              <resp-connector cache="example" /> 
              <rest-connector />
            </endpoint>
          </endpoints>
        </server>
    </infinispan>

Root Cause

DG Operator introduces the ConfigMap usage for adding advanced customization capabilities.
However, for simple JVM flags, user can set extraJvmOpts.
Below is an example with both: spec.configMapName and spec.container.extraJvmOpts: 

  spec:
    configMapName: "${CLUSTER_NAME}-custom-config"  <------------------------ that's the config map
    container:
      cpu: '2'
      extraJvmOpts: '-Xlog:gc*=info:file=/tmp/gc.log:time,level,tags,uptimemillis:filecount=10,filesize=1m
              -Dcom.redhat.fips=false -Duser.timezone=Europe/Madrid'

However for Metaspace/MaxMetaspace, that's not possible Can you customize Metaspace and MaxMetaspace in DG as JVM argument in DG Operator?.

Customizing resource on Operator pod itself

For details see the solution Customizing resource on DG Operator pod itself in OCP 4. In a simplified manner, for customizing the Operator itself (the pod operator) the values must be set on the OLM subscription - in case of issues, like prevention of leader election timing out:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata: 
  name: datagrid
spec: 
  channel: 8.3.x
  installPlanApproval: Manual
  name: datagrid
  source: redhat-operators
  sourceNamespace: openshift-marketplace
  startingCSV: datagrid-operator.v8.3.6
  config: 
    resources: 
      requests: 
        memory: "64Mi"
        cpu: "250m"
      limits: 
        memory: "128Mi"
        cpu: "500m"

Increasing the DG Operator resources can be useful when many Infinispan Custom Resources are being handled by the Operator, e.g. 100/200+ Caches/Infinispan CRs. So then the leader might timeout as above, and editing the Subscription can be used to increase its resources directly without editing/changing the deployment, which will be overwritten by the OLM.

Adding custom label and custom annotations

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  annotations:
    infinispan.org/monitoring: 'true'
    infinispan.org/operatorPodTargetLabels: >-
      com.redhat.component-name,com.redhat.component-type,com.redhat.component-version,com.redhat.product-name,com.redhat.product-version,fruit
  namespace: dg-test-1
  labels:
    com.redhat.component-name: Data_Grid
...
    fruit: apple

Example using fruit: apple annotation:

### Infinispan CR:
$ oc get infinispan example-infinispan -o yaml | grep fruit
    infinispan.org/operatorPodTargetLabels: com.redhat.component-name,com.redhat.component-type,com.redhat.component-version,com.redhat.product-name,com.redhat.product-version,fruit
    fruit: apple
### get pod:
$ oc get pod
NAME                                                      READY   STATUS    RESTARTS   AGE
example-infinispan-0                                      1/1     Running   0          34s <----
example-infinispan-config-listener-9c44f6c6d-9t6db        1/1     Running   0          17s
infinispan-operator-controller-manager-7c68cd4668-2gpz8   1/1     Running   0          99m
### pod yaml::
$ oc get pod example-infinispan-0 -o yaml | grep fruit
    fruit: apple

Adding environment variables

This example is helpful in case you need to make server data dir writable. For instance in the case of MAKE_DATADIR_WRITABLE, see below:

apiVersion: infinispan.org/v1
kind: Infinispan
metadata:
  annotations:
    infinispan.org/monitoring: 'true'
    infinispan.org/operatorPodTargetLabels: >-
      com.redhat.component-name,com.redhat.component-type,com.redhat.component-version,com.redhat.product-name,com.redhat.product-version,fruit
  namespace: dg-test-1
spec:
  config:
    env:
    - name: MAKE_DATADIR_WRITABLE
      value: "true"
PurposeSolutions
Metaspace settings (via Subscription)Can you customize Metaspace and MaxMetaspace in DG as JVM argument in DG Operator?
Setting DG Operator pod settingsCustomizing resource on DG Operator pod itself in OCP 4
Print DG Operand configurationHow to print the configuration used by DG Operator at startup?
DG 8 connection via CLIConnecting on DG 8 CLI inside OCP 4 pod/terminal

Diagnostic Steps

  1. To calculate the number of non-blocking threads - run the script below (attached as local_verification.sh:
echo "Calculate the number of non blocking threads and cpus"
nonblockingpid=$(jcmd | awk '$3=="org.infinispan.server.Bootstrap" {print $1}')
nthreads=$(jcmd $nonblockingpid Thread.print | grep "non-blocking-thread" | wc -l)
echo "number of non blocking threads == " $nthreads
mythreads=$(jcmd $nonblockingpid Thread.print | grep "mythreads" | wc -l)
echo "number of < my threads> == " $mythreads
bthreads=$(jcmd $nonblockingpid Thread.print | grep '"blocking-thread.' | wc -l)
echo "number of blocking threads == " $bthreads
jgroupsthreads=$(jcmd $nonblockingpid Thread.print | grep '"jgroups.' | wc -l)
echo "number of jgroups threads == " $jgroupsthreads
ncpus=$(lscpu | grep "CPU(s)" | awk 'NR==1 {print $2}')
echo "number of cpus: == " $ncpus

Example:

$ ./local_verification.sh 
Calculate the number of non blocking threads and cpus
number of non blocking threads ==  57
number of < my threads> ==  0
number of blocking threads ==  2
number of jgroups threads ==  50
number of cpus: ==  8

Printing the runtime configuration:

spec:
  logging:
    categories:
      org.infinispan.SERVER: debug <------------ print the configuration
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.