Problems configuring KUBE_PING in JBoss EAP

Solution Verified - Updated

Environment

  • Openshift
    • 3.x
    • 4.x <--- EAP Operator
  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.4
    • 7.x
  • Red Hat Single Sign-On (RH-SSO)
    • 7.x

Issue

We receive the following warning in RH-SSO pod logs:

10:23:47,690 WARN  [org.jgroups.protocols.kubernetes.KUBE_PING] (thread-207,ejb,sso-39-f8pjp) failed getting JSON response from Kubernetes Client[masterUrl=https://10.10.10.10:443/api/v1 , headers={Authorization=#MASKED:835#}, connectTimeout=5000, readTimeout=30000, operationAttempts=3, operationSleep=1000, streamProvider=org.jgroups.protocols.kubernetes.stream.TokenStreamProvider@74333efa] for cluster [ejb], 
namespace [default], labels [null]; encountered [java.lang.Exception: 3 attempt(s) with a 1000ms sleep to execute [OpenStream] failed. Last failure was [java.io.IOException: Server returned HTTP response code: 403 for URL: https://10.10.10.10:443/api/v1/namespaces/default/pods ]]

Resolution

This solution applies specifically for issues with KUBE_PING, for other details on implementing/disabling/default clustering behavior see solution EAP 7 image clustering in OCP 4.

Issue: I don't want my JBoss EAP (or RH-SSO) instances to work as a cluster (disable KUBE_PING)

The OPENSHIFT_KUBE_PING_NAMESPACE environment variable must be set, and in this case, the server will act as if it is a single-node cluster (a "cluster of one")

In case the application does not require cluster communication and the KUBE_PING needs to be disabled, it will be enough to just remove the default environment variable OPENSHIFT_KUBE_PING_NAMESPACE from the BuildConfig and DeploymentConfig

After that, a new build and deployment will take place. During the JBoss EAP start, the following warning will be displayed once:

WARNING: Environment variable OPENSHIFT_KUBE_PING_NAMESPACE undefined. Clustering will be unavailable. Please refer to the documentation for configuration.

There is no way to remove clustering capabilities from the image without changing the standalone-openshift.xml, the trick above only makes it not build the cluster. It doesn't remove the clustering capabilities.

Issue: WARNING: Service account has insufficient permissions to view pods in kubernetes

WARNING: Service account has insufficient permissions to view pods in kubernetes (HTTP 403). Clustering will be unavailable. Please refer to the documentation for configuration.
  ... more logs ...
  16:43:27,070 WARN  [org.jgroups.protocols.openshift.KUBE_PING] (ServerService Thread Pool -- 54) Problem getting Pod json from Kubernetes Client[masterUrl=https://10.10.10.10:443/api/v1, headers{Authorization=#MASKED:855#}, connectTimeout=5000, readTimeout=30000, operationAttempts=3, operationSleep=1000, streamProvider=org.openshift.ping.common.stream.InsecureStreamProvider@61f84f09] for cluster [hornetq-channel], namespace [test-eap], labels [app=jboss-eap64-openshift]; encountered [java.lang.Exception: 3 attempt(s) with a 1000ms sleep to execute [OpenStream] failed. Last failure was [java.io.IOException: Server returned HTTP response code: 403 for URL: https://10.10.10.10:443/api/v1/namespaces/test-eap/pods?labelSelector=app%3Djboss-eap64-openshift]]

If this error appears periodically it means that the service account needs to be granted some additional permission. This should be fixed by adding the view role to the EAP service account:

$ oc policy add-role-to-user view -z eap-service-account

Note: For RH-SSO, this should be like:

$ oc policy add-role-to-user view system:serviceaccount:$(oc project -q):default

Issue: HttpConnections are not closed when using KUBE_PING (only JBoss EAP 6.4)

When using JBoss EAP 6.4 version 1.4.x or before there is a known bug tracked with this This content is not included.JIRA Ticket - CLOUD-1461
This issue has been solved in openshift-ping-kube-1.1.2.Final-redhat-1.jar that will be included in JBoss EAP64 1.5.0.GA

This problem is only manifested in EAP6, since Kube Ping uses the JBossServer implementation, whereas in EAP7 it uses the UndertowServer implementation.

Issue: "WFLYCLJG0016: Unable to load protocol class org.jgroups.protocols.openshift.DNS_PING

If this value were configured by using template/environment variables, the launch scripts changed it to valid protocols, like dns.DNS_PING. However, the user can replace the standalone-openshift.xml to use the old protocols and they won't be loaded and the issue will occur.

For more information about JBossEAP Clustering in Openshift refer to the This content is not included.JBoss EAP for Openshift documentation.

Root Cause

The default behavior is the clustering behavior (using kubernetes.KUBE_PING - replacing openshift.DNS_PING as explained on here) which requires additional service account permissions as mentioned in the docs.
In the templates it is set default to JGROUPS_PING_PROTOCOL=dns.DNS_PING, which doesn't require additional permissions.
One can set OPENSHIFT_DNS_PING_SERVICE_NAME to something unique and not have a cluster. (Clustering is still enabled, but each pod / deployment will not see the other members.)
There is no way to remove clustering capabilities from the image without changing the standalone-openshift.xml, the usage of OPENSHIFT_KUBE_PING_NAMESPACE for a random value only makes it not build the cluster. It doesn't remove the clustering capabilities, for removing capabilities one must use Galleon layers.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.