Gossip Router pod in DG 8 OCP 4

Environment

• OpenJDK 11
• Red hat OpenShift Container Platform (OCP)
  • 4.x
• Red Hat Data Grid (RHDG)
  • 8.x

Issue

• What is the Gossip Router pod?
• How much memory it has?
• What are the message it carries?

Resolution

DG Operator 8.3.x introduced the Xsite exchange of data via GossipRouter pod via tunnel protocol. Each site will have one, but JGroups only needs a single GossipRouter and if one of the sites crashes, it uses the other site's GossipRouter.
Following the diagram shows Tunnel protocol usage:

custer1Node -> cluster1Master -> GossipRouter -> cluster2Master -> cluster2OtherNode
which is the same as:
cluster1Node -> cluster1Relay1 -> GossipRouter -> cluster1Relay2 -> cluster2OtherNode

Whereas without TUNNEL just remove the step in the middle,

custer1Node -> cluster1Master -> cluster2Master -> cluster2OtherNode

Dimensioning the Gossip Router pod

Gossip Router pod is unbounded so it should use the limit set on the limit-range setting on the environment.
Depending on the environment (and load) it should be recommended to set the size as 256mb or even 512mb. Particularly given that the heap will have the SSL Impl objects from the OCP 4 probes, see below.

Tunnel protocol

Introduced together with Gossip Router is the Tunnel protocol, it is an open source project and can be accessed Content from www.jgroups.org is not included.here, for tunnel information see Content from jgroups.org is not included.Tunnel Protocol
The tunnel protocol doesn't have a timeout or heartbeat set currently

Default GC collector and JVM tunning

The default collector is SerialGC, as explained on Deprecated service type Cache in DG 8 in OCP 4 for GR (and the other pods as well), that's because the pod will use the default pod and it has one cpu, so the JVM will set the default collector for SerialGC.
For adding jvm flags (starting on DG Operator 8.3.7) set them via spec.container.routerExtraJvmOpts (lower case r):

routerExtraJvmOpts: '-XX:MaxDirectMemorySize=4M -Xlog:gc*=info:file=/tmp/gc.log:time,level,tags,uptimemillis:filecount=10,filesize=1m'

OCP 4 probes frequent ping

As any pod, Openshift will keep sending pings to see if the pod is alive (liveness probe) or ready (readines probe) for the Gossip Router pod, set for 10s period:

Meaning those probes can be set either at the container or the pod level that either run a script to see if the application is running (liveness) or they run a curl to see if the pod is accepting traffic (readiness). And they have thresholds and timeouts, etc, and if n number of probes fails (this set on the pod configuration), the pod is restarted by the kubelet - so this is important for the operation for pod utilization.
Particularly the readiness probe, it will do a curl on the connection probe sending pings to the Gossip Router pod, i.e. the liveness/readness probe that send a ping to the Gossip router pod - sending one each 10s (6 per minute/ so 360/h) will mean a SSL connection will open for each one. Those SSL connections might require a full GC to clean them up, since they use finalizers.

SSL connections on heap

Considering the above, ie. SSL connections caused by the readness probs, those SSL will accumulate and will take a Full GC to be cleaned and some collectors avoid full GCs, e.g G1GC. So then the Gossip router pod, which is unbounded (so it will take the limit range by default), must take in consideration those SSL connections for its size.

Warning on the Gossip router pod logs

Because of the probe, the Gossip Router will ping as below: SSLHandshakeException - every 10s or so:

Aug 22, 2022 5:13:07 PM org.jgroups.blocks.cs.TcpConnection$Receiver run WARNING: failed handling message javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1696) at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1514) at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1416) at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) at java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:921) at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:1012) at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:271) at java.base/java.io.DataInputStream.readInt(DataInputStream.java:392) at org.jgroups.blocks.cs.TcpConnection$Receiver.run(TcpConnection.java:305Send to:Everyone  

Root Cause

Gossip Router pod was introduced on Dg Operator 8.3.x see solution Differences between DG 8.3 vs DG 8.2 for more differences on DG 8.3.x vs DG 8.2.x.

Gossip router process

The gossip router pod is a pod that has is executing jgroups' jar that executes a main function, which is the gossip router: org.jgroups.stack.GossipRouter:

java -cp /opt/infinispan/lib/jgroups-5.2.12.Final-redhat-00001.jar ... org.jgroups.stack.GossipRouter -port 7900 -dump_msgs registration

The org.jgroups.stackGossipRouter will have:

public class GossipRouter extends org.jgroups.blocks.cs.ReceiverAdapter implements <-----
org.jgroups.blocks.cs.ConnectionListener, org.jgroups.stack.DiagnosticsHandler.ProbeHandler 

Example DG 8 Operator configuration:

    <stack name="relay-tunnel" extends="udp">
        <TUNNEL
            bind_addr="${jgroups.relay.bind.address:SITE_LOCAL}"
            bind_port="${jgroups.relay.bind.port:0}"
            gossip_router_hosts="dg-cluster-nyc-site[7900],dg-cluster-lon-site.dg-test-lon.svc.cluster.local[7900]"
            enable_diagnostics="false"
            port_range="0"
            stack.combine="REPLACE"
            stack.position="UDP"
        />
        <!-- we are unable to use FD_SOCK with openshift -->
        <!-- otherwise, we would need 1 external service per pod -->
        <FD_SOCK stack.combine="REMOVE"/>
    </stack>

The example above has two sites for configuration and the pods must all have the same GR list. If one connects to a GR all others aren't also connected to then they will not communicate correctly. Picking one site's GR and everyone using it would work, until that site went down. If only 2 sites then that should work.

Truststore and Cross-site

The trust store must contain the certificates you created (and/or the CA authority that signed them) and configured in the transport and router key stores - and must be found by the Router.

Diagnostic Steps

For Cross-Site/jgroups investigation enable jgroups tracing logs:

spec:
  logging:
    categories:
      org.jgroups: trace

Related

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.