EAP 7 image clustering in OCP 4

Solution Verified - Updated 26 Nov 2024

Environment

Red Hat JBoss Enterprise Application Platform (EAP)
- 7.x
Red hat OpenShift Container Platform (OCP)
- 4.x

Issue

How to set EAP 7 clustering in OCP 4?
How to disable EAP 7 clustering in OCP 4?

Resolution

EAP 7 images have some environment variables (set on template deployment) that control clustering behavior - the image's behavior changes according to those environment variables.
The most important is the JGROUPS_PING_PROTOCOL, which will set the discovery jgroups protocol: DNS_PING or KUBE_PING accordingly (or LOCAL_PING).

There are two requirements for clustering to work:

Clustering subsystems (jgroups) present in the configuration (attention the dynamic changes it happens with the loading scripts)
The correct env variable set, the actual variables will depend on the type clustering set: KUBE_PING or DNS_PING.

JGroups Discovery Mechanisms

There are two mechanisms for enabling clustering on the EAP images: DNS_PING and KUBE_PING (kubernetes.KUBE_PING replaced openshift.KUBE_PING). For setting it up:

Protocol	Setting
for `KUBE_PING`	use `JGROUPS_PING_PROTOCOL=kubernetes.KUBE_PING`
for `DNS_PING`	use `JGROUPS_PING_PROTOCOL=dns.DNS_PING`
for `LOCAL_PING`	use `JGROUPS_PING_PROTOCOL=LOCAL_PING`

As highlighted above, another possibility is to set LOCAL_PING as the discovery protocol that will find members on the local process only. This can be an alternative for disabling clustering very easily setting a unique environment variable.

Enabling clustering in EAP 7 images

Discovery Protocol	Enviroment variables and task	Service Account	Permissions
DNS_PING	set `JGROUPS_PING_PROTOCOL=dns.DNS_PING`,`OPENSHIFT_DNS_PING_SERVICE_NAME`, `OPENSHIFT_DNS_PING_SERVICE_PORT`, and a service to expose it	works with default permissions	No need for SA modication
KUBE_PING	`JGROUPS_PING_PROTOCOL=kubernetes.KUBE_PING`, `KUBERNETES_NAMESPACE=$namespace`, `KUBERNETES_LABELS=application`, account authorization	need more permissions	SA modication

Template default discovery mechanism

When deploying via template, the template will set dns.DNS_PING, so on the logs there will be:

INFO Configuring JGroups discovery protocol to dns.DNS_PING

The clustering capabilities come by default on template deployment for dns.DNS_PING.
Note: DNS_PING is the only protocol supported by template.

Requirements for DNS_PING

To set DNS_PING it is required, an environment variable JGROUPS_PING_PROTOCOL set for dns.DNS_PING, the variable OPENSHIFT_DNS_PING_SERVICE_NAME for the right service, and the variable OPENSHIFT_DNS_PING_SERVICE_PORT for the right port.
Finally it requires a service to expose it, this service will be This page is not included, but the link has been rewritten to point to the nearest parent document.headless meaning it does not have ip and consequently ping directly on the service will fail.
For DNS there is no need for extra authorization, so that's the reason templates have it as default.
To disable, set OPENSHIFT_DNS_PING_SERVICE_PORT for another port, or remove the service. This will disable DNS_PING protocol.
On DNS_PING, there are two ports that come into play but have different purposes: OPENSHIFT_DNS_PING_SERVICE_PORT (8888) and TCP/UDP port (7600). By default 8888 is for the DNS lookup to the ping-service (default eap-app-ping service), and all cluster communication are on 7600 between EAP/RHDG instances.

Port	Purpose
8888	The DNS lookup to the ping-service
7600	TCP communication

Requirements for `KUBE_PING`

The environment variable OPENSHIFT_KUBE_PING_NAMESPACE must be defined properly - on the deployment/deployment config or customer resource accordingly (KUBE_PING might set for OPENSHIFT_KUBE_PING_NAMESPACE for default namespace, so avoid deploying applications on default namespace for non-wanted clustering behavior).
The corollary of the above requirement (for clustering) is that if one does not OPENSHIFT_KUBE_PING_NAMESPACE defined one will not have clustering capabilities.
Therefore one procedure do disable clustering is to set other value for OPENSHIFT_KUBE_PING_NAMESPACE for another unrelated label. Given it requires extra permissions, so then it might occur in Service account has insufficient permissions to view pods in kubernetes. For more information on Content from github.com is not included.KUBE_PING.

Disabling clustering

The default template, which sets discovery protocol for DNS_PING, makes the EAP 7 image clustered by default and there are a few options for disabling:

Protocol	How to disable it	Implication
DNS_PING	Remove service associated with `OPENSHIFT_DNS_PING_SERVICE_NAME`, or set `OPENSHIFT_DNS_PING_SERVICE_PORT` for another port	Jgroups still there
KUBE_PING	KUBERNETES_NAMESPACE for another namespace	Jgroups still there
Both	Edit `standalone-openshift.xml` and remove the jgroups	Need to overwrite the default configuration or use S2I - need to remove the other subsystems
Both	Set `LOCAL_PING` as discovery protocol	`JGROUPS_PING_PROTOCOL=LOCAL_PING` will discover members on same process

As mentioned above, an option could be to remove those capabilities from the standalone-openshift.xml and overwrite the default file, however this procedure most of the times can be not optimal (given the process itself).

Default Jgroups stack (vanilla image default)

In EAP 7 OCP images, the Jgroups stack has by default clustering capabilities defined on both stacks tcp and udp via kubernetes.KUBE_PING protocol or openshift.KUBE_PING(according to the version, openshift.KUBE_PING was deprecated for kubernetes.KUBE_PING):

   419                <stacks>
   420                    <stack name="tcp">
   421                        <transport type="TCP" socket-binding="jgroups-tcp"/>
   422                        <protocol type="kubernetes.KUBE_PING"/>
   423                        <protocol type="MERGE3"/>
   424                        <protocol type="FD_SOCK"/>
   425                        <protocol type="FD_ALL"/>

The implication of the default being KUBE_PING is that the template deployment requires cli commands at boot time, which can be seen and the hint to verify the definition of OPENSHIFT_KUBE_PING_NAMESPACE is at start up script - this is not the server.log - given the EAP 7 did not start yet. The start scripts will kick in first, and then the EAP process start:

23:36:02,867 INFO  [org.jboss.as.controller.management-deprecated] (CLI command executor) WFLYCTL0395: Operation add against the resource at address /subsystem=jgroups/stack=tcp/protocol=dns.DNS_PING/property=dns_query is deprecated, and it might be removed in future version. See the the output of the read-operation-description operation to learn more about the deprecation.
....

Another implication of using KUBE_PING and not defining the namespace nor labels are set, then the IP addresses of all pods created by Kubernetes in the default namespace default will be returned. So this will make a default clustering behavior even without defining OPENSHIFT_KUBE_PING_NAMESPACE - see below:

WARN Environment variable OPENSHIFT_KUBE_PING_NAMESPACE undefined. Clustering will be unavailable. Please refer to the documentation for configuration.
INFO Configuring JGroups discovery protocol to kubernetes.KUBE_PING
INFO  [org.jgroups.protocols.kubernetes.KUBE_PING] (ServerService Thread Pool -- 73) namespace default set; clustering enabled

Logging verification

kubernetes.KUBE_PING usage - clustering enabled:

$ oc logs $pods
INFO Configuring JGroups cluster traffic encryption protocol to SYM_ENCRYPT.
WARN Detected missing JGroups encryption configuration, the communication within the cluster WILL NOT be encrypted.
WARN No password defined for JGroups cluster. AUTH protocol will be disabled. Please define JGROUPS_CLUSTER_PASSWORD.
WARN Service account has insufficient permissions to view pods in kubernetes (HTTP 403). Clustering might be unavailable. Please refer to the documentation for configuration.
INFO Configuring JGroups discovery protocol to kubernetes.KUBE_PING <-------------------------------------------------
INFO Using PicketBox SSL configuration.
INFO Access log is disabled, ignoring configuration.

Disabled clustering:

$ oc logs $pods
INFO Configuring JGroups cluster traffic encryption protocol to SYM_ENCRYPT.
WARN Detected missing JGroups encryption configuration, the communication within the cluster WILL NOT be encrypted.
WARN No password defined for JGroups cluster. AUTH protocol will be disabled. Please define JGROUPS_CLUSTER_PASSWORD.
WARN Environment variable OPENSHIFT_KUBE_PING_NAMESPACE undefined. Clustering will be unavailable.  <-----------------------------------
Please refer to the documentation for configuration. <--------------------------------
INFO Configuring JGroups discovery protocol to kubernetes.KUBE_PING 
INFO Using PicketBox SSL configuration.
INFO Access log is disabled, ignoring configuration.

Deprecated protocols

Deprecated protocol	Current protocol
openshift.KUBE_PING	kubernetes.KUBE_PING
openshift.DNS_PING	dns.DNS_PING

The usage of deprecated protocol result in WFLYCLJG0016 - as below:

WFLYCLJG0016: Unable to load protocol class org.jgroups.protocols.openshift.DNS_PING" error.

Automatic addition of jgroups protocols

If jgroups is in the config, the server will add protocols, for instance KUBE_PING. That's because the image must have a discovery protocol in the stack, and a CLI script is run to add it - the default scripts do this (i.e. bin/launch/ha.sh).
This is not done by the parser, but some other protocol (GMS) expect a discovery protocol to be there. And this is done at runtime not build time - meaning right before starting the KUBE_PING will be added even if LOCAL_PING is used.
This can be disabled with setting CONFIG_IS_FINAL for true, CONFIG_IS_FINAL=true. Given a disabled automatic configuration setting, one can set LOCAL_PING instead of KUBE_PING to disable clustering. And not CONFIG_IS_FINAL=1.

Variable	Purpose	Options
`CONFIG_IS_FINAL`	Makes the configuration final	Options are true/false

EAP Operator

EAP Operator deployment, when deploying the EAP image will set the default KUBE_PING protocol.
And therefore requires adequate service account permissions - otherwise EAP 7 won't be clustered and the image will return (at the top):

WARN Service account has insufficient permissions to view pods in kubernetes (HTTP 403). Clustering might be unavailable. Please refer to the documentation for configuration.

Deployment method	jgroups discover
EAP Operator	KUBE_PING (default); DNS_PING (setting env variables and creating an eap-app-ping headless service)
EAP Template	DNS_PING (automatically comes with eap-app-ping), KUBE_PING not on default templates

Galleon layers

Another alternative for avoiding EAP 7 images with clustering capabilities altogether, is to use to provision a trimmed server (using Galleon layers) that doesn't contain clustering. This is discussed on JBoss EAP 7 thin image.

Testing clustering

The clustering capabilities doe not start unless there's an application deployed that needs it.
And consequently, the clustering logs, e.g. ISPN000078/ISPN000094 only come after the deployment clustered applications:

$ oc logs eap-example-0 | grep ISPN000094
02:28:47,519 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 14) ISPN000094: Received new cluster view for channel ee: [eap-example-0|0] (1) [eap-example-0]

Root Cause

FAQ

Q1. How to enable clustering in EAP 7?
A1. Clustering is enabled by default on template deployment. For other deployments, set the environment variables depending on DNS_PING or KUBE_PING usage. Together with the clustering subsystem present in the configuration, the environment variables will set the clustering capabilities.

Q2. How to disable clustering (in default image deployment)?
A2. EAP 7 image should bring KUBE_PING by default, using the protocol LOCAL_PING can disable it - this can be done via a custom config that requires CONFIG_IS_FINAL=true or keep the standalone-openshift.xml of the server and simply set the environment variable: JGROUPS_PING_PROTOCOL=LOCAL_PING. The easiest is to remove the discovery service remove the service that is used by the EAP's discovery protocol to make the comms/initial discovery, so just remove it and the pods won't find one another.

Q3. What are options for overwriting/using a custom standalone-openshift.xml?
A3. See options on solutions What are the options to use a custom runtime JBoss EAP 7 configurations in OCP and custom standalone-openshift.xml. Remember to use CONFIG_IS_FINAL as detailed on section Automatic addition of jgroups protocols.

Q4. What is the default jgroups stack for EAP 7?
A4. Below:

   419                <stacks>
   420                    <stack name="tcp">
   421                        <transport type="TCP" socket-binding="jgroups-tcp"/>
   422                        <protocol type="kubernetes.KUBE_PING"/>
   423                        <protocol type="MERGE3"/>
   424                        <protocol type="FD_SOCK"/>
   425                        <protocol type="FD_ALL"/>

Q5. The EAP 7 image has KUBE_PING and OPENSHIFT_KUBE_PING_NAMESPACE is not set, why it is clustering?
A5. See the start logs if the OPENSHIFT_KUBE_PING_NAMESPACE is not being set for default namespace. KUBE_PING had default set as namespaces, Content from github.com is not included.here.

Q6. We are overwriting KUBE_PING on the protocol stack but somehow the protocol is making the come back to the configuration
A6. See section Automatic addition of jgroups protocols. This is expected if CONFIG_IS_FINAL is not set (this is done by starting scripts, i.e. bin/launch/ha.sh) - so the jgroups is detected and KUBE_PING will be added:

<stacks>
                <stack name="tcp">
                    <transport type="TCP" socket-binding="jgroups-tcp"/>
                    <protocol type="kubernetes.KUBE_PING"/> <!--- this will be added -->
                    <protocol type="LOCAL_PING"/> <!-- even though this was set on the custom configuration -->
                    <protocol type="MERGE3"/>
                    <protocol type="FD_SOCK"/>
                    <protocol type="FD_ALL"/>
                    <protocol type="VERIFY_SUSPECT"/>
                    <protocol type="pbcast.NAKACK2"/>
                    <protocol type="UNICAST3"/>
                    <protocol type="pbcast.STABLE"/>
                    <auth-protocol type="AUTH">
                        <digest-token algorithm="SHA-512">
                            <shared-secret-reference clear-text="mh1WUIbK"/>
                        </digest-token>
                    </auth-protocol>
                    <protocol type="pbcast.GMS"/>
                    <protocol type="UFC"/>
                    <protocol type="MFC"/>
                    <protocol type="FRAG3"/>
                </stack>

Diagnostic Steps

From the pod logs (which will be on the inspect):

The template deployment of EAP 7 will use dns.DNS_PING by default - on the pod logs see:

INFO Configuring JGroups discovery protocol to dns.DNS_PING

When setting kube_ping see the message as below:

INFO Configuring JGroups discovery protocol to kubernetes.KUBE_PING <-------------------------------------------------

Clustering will start when a application needs it start - search for ISPN000078 and ISPN000094:

02:11:01,120 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 16) ISPN000094: Received new cluster view for channel ee: [eap-app-1-2qswv|3] (4) [eap-app-1-2qswv, eap-app-1-57hdw, eap-app-1-gpt76, eap-app-1-sp8hs]

Trace logging

To enable tracing logging on DNS_PING: org.jgroups.protocols.DNS_PING = TRACE. See solution EAP 7 DNS_PING tracing logs in OCP 4 for interpretation of tracing logs.

Solution	Purpose
EAP 7 clustering deployments boundaries in OCP cluster	EAP 7 service details
EAP 7 DNS_PING tracing logs in OCP 4	DNS_PING tracing logging enablement and interpretation

SBR

JBoss Base AS

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Learn more

Tags

eap
jboss

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.