Data Grid 8 Operator Clustering details

Environment

• Red hat OpenShift Container Platform (OCP)
  • 4.x
• Red Hat Data Grid (RHDG)
  • 8.x

Issue

• How does Data Grid 8 clustering works?
• Which services DG 8 relies on?

Resolution

Data Grid clustering relies on DNS_PING jgroups discovery protocol, which is explained on: EAP 7 image clustering in OCP 4, see below:

22:46:44,633 DEBUG (jgroups-6,example-infinispan-0-8160) [org.jgroups.protocols.dns.DNS_PING] example-infinispan-0-8160: entries collected from DNS (in 0 ms): [127.0.0.0:0, 128.0.0.0:0]

Data Grid 8 is not based on environment variables such as JDG 7 or EAP 7 - the clustering settings come from the DG Operator, which parses the Custom Resources.

Data Grid configuration:

### Configuration
sh-4.4$ cat /opt/infinispan/server/conf/operator/infinispan-admin.xml 
<infinispan
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="urn:infinispan:config:14.0 https://infinispan.org/schemas/infinispan-config-14.0.xsd
                        urn:infinispan:server:14.0 https://infinispan.org/schemas/infinispan-server-14.0.xsd
                        urn:org:jgroups http://www.jgroups.org/schema/jgroups-5.2.xsd"
    xmlns="urn:infinispan:config:14.0"
    xmlns:server="urn:infinispan:server:14.0">
<jgroups>
    <stack name="image-tcp" extends="tcp">
        <TCP bind_addr="${jgroups.bind.address:SITE_LOCAL}"
             bind_port="${jgroups.bind.port,jgroups.tcp.port:7800}"
             diag.enabled="false"
             port_range="0"/>
        <dns.DNS_PING dns_query="example-ping.dg-test-nyc.svc.cluster.local"
                      dns_record_type="A"
                      stack.combine="REPLACE" stack.position="MPING"/>
    </stack>

In Data Grid the selector is based on two labels app: infinispan-pod and clusterName: $infinispan_name. This will make the infinispan's cluster distinguishable. Meaning each cluster will have a unique clusterName, hence how it distinguishes the pods. User cannot have cluster's with the same name on the same namespace.

Example:

Deploying an Infinispan Cr named example-infinispan and setting it up with two pods, creates the below services and endpoints:
Services

### services:
$ oc get service -o wide
NAME                                             TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)     AGE    SELECTOR
example-infinispan                               ClusterIP   127.0.0.4    <none>        11222/TCP   50m    app=infinispan-pod,clusterName=example-infinispan
example-infinispan-admin                         ClusterIP   None            <none>        11223/TCP   50m    app=infinispan-pod,clusterName=example-infinispan
example-infinispan-ping                          ClusterIP   None            <none>        8888/TCP    50m    app=infinispan-pod,clusterName=example-infinispan
infinispan-operator-controller-manager-service   ClusterIP   127.0.0.5   <none>        443/TCP     32m    app.kubernetes.io/name=infinispan-operator,control-plane=controller-manager

Enpoints:

### endpoints:
$ oc get endpoints -w
NAME                                             ENDPOINTS                             AGE
example-infinispan                               127.0.0.0.1:11222,127.0.0.0.2:11222   25m <---- endpoints are the two pods
example-infinispan-admin                         127.0.0.0.1:11223,127.0.0.0.2:11223   25m <---- endpoints are the two pods
example-infinispan-ping                          127.0.0.0.1:8888,127.0.0.0.1::8888     25m <---- endpoints are the two pods
infinispan-operator-controller-manager-service   127.0.0.0.3:9443                      8m11s

Service table:

The example-infinispan* (internal service) is the one to be accessed by other applications within OCP (even different namespaces). As explained on Data Grid 8 Operator Exposition Route vs NodePort vs LoadBalancer in Openshift 4.

Pods

$ oc get pod -o wide
NAME                                                      READY   STATUS    RESTARTS   AGE   IP            NODE                         
example-infinispan-0                                      1/1     Running   0          48m   127.0.0.0.1   ip-10-0-139-7.ec2.internal   
example-infinispan-1                                      1/1     Running   0          23m   126.0.0.0.2   ip-10-0-139-7.ec2.internal   
example-infinispan-config-listener-id                     1/1     Running   0          48m   127.0.0.0.6  ip-10-0-139-7.ec2.internal
infinispan-operator-controller-manager-id                 1/1     Running   0          31m   128.0.0.0.7   ip-10-0-139-7.ec2.internal

Above shows the infinispan-operator-controller-manager-service, however, if the Operator is deployed in another namespace, the infinispan-operator-controller-manager-service will be in this namespace. For example if cluster-wide Operator installation, the operator pod (controller manager) and the service will be on openshift-operators namespace.

Root Cause

Data Grid Operator will create those four services by the operator itself (meaning the operator creates a deployment yaml with this service specs) - including metadata and spec section.
The user should not try to change those specs, otherwise they will be overwritten by the Operator.

Clustering and multicast:

In OCP the clustering capabilities, are provided by the multicast channel not by the tcp stack.
Therefore, if the configuration recommends to use tcp stack, which should not be used on the ocp, and the MPING opens a new channel which doesn't merge the pods.

    infinispan-config.yaml: >
      infinispan:
        cache-container:
...
          transport:
            stack: "tcp" <------------------- will prevent the merge of the pods

publishNoReadyAddresses

Data Grid Operator (before 8.4.5) and Helm charts (initially) had an issue with the DG pods service not having the spec.publishNotReadyAddresses, which is required by jgroups to cluster properly the DG pods. Full explanation here.

$ oc get svc infinispan-ping -o yaml
apiVersion: v1
kind: Service
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: infinispan
    port: 8888
    protocol: TCP
    targetPort: 8888
  publishNotReadyAddresses: true <------------------ 
  selector:
    app: infinispan-pod
    clusterName: infinispan
  sessionAffinity: None
  type: ClusterIP

Configuration

DG in OCP will be a combination of the three following files, where the first two are autogenerated by the operator: infinispan-base.xml + infinispan-admin.xml + infinispan-config.xml

Issues:

This content is not included.[Operator] Set publishNotReadyAddresses=true on ping service: The ping service don't have publishNotReadyAddresses

Diagnostic Steps

1. Do getent hosts $SVC to return the host:

$ getent hosts dg-cluster-nyc.dg-test-nyc.svc.cluster.local
127.0.0.1  dg-cluster-nyc.dg-test-nyc.svc.cluster.local

2. Setting debug logging on the Infinispan CR:

spec:
logging:
  categories:
    org.infinispan: debug
    org.jgroups: debug

To investigate endpoints and services, get the inspect Using inspect for DG 8 troubleshooting, which will have pod logs, service yaml, endpoints yaml as well.

3. Verify the dns_query's default service name:

sh-4.4$ cat /opt/infinispan/server/conf/operator/infinispan-admin.xml  | grep dns_query
        <dns.DNS_PING dns_query="infinispan_cr-ping.dg-test.svc.cluster.local"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.