EAP 7 DNS_PING tracing logs in OCP 4

Solution Verified - Updated 10 Sept 2025

Environment

Red Hat JBoss Enterprise Application Platform (EAP)
- 7.x
Red hat OpenShift Container Platform (OCP)
- 4.x

Issue

How to enable DNS_PING tracing logs in EAP 7 in OCP 4?
How to interpret DNS_PING's tracing logs?

Resolution

The solution EAP 7 image clustering in OCP 4 bring several aspects of clustering in EAP 7.

For enabling DNS_PING tracing logging

What are the options to use a custom runtime JBoss EAP 7 configurations in OCP.

Set org.jgroups.protocols.dns with level TRACE so the complete list of operations will appear. DNS_PING trace logs, will not bring jgroups' protocols output. The recommendation is to org.jgroups.protocols.DNS_PING and org.jgroups:TRACE.
Example:

# set LOGGER_CATEGORIES for org.jgroups.protocols.DNS_PING:TRACE org.jgroups:TRACE
$ oc set env dc/eap-app LOGGER_CATEGORIES=org.jgroups.protocols.DNS_PING:TRACE,org.jgroups:TRACE

^ Above will be an update on the deployment(config) and terminate the pod and recreate new pods.
Note that setting org.jgroups.protocols.DNS_PING with level TRACE won't show the GET_MBRS_REQ or GET_MBRS_RSP messages.

Process

dns.DNS_PING: set the discovery protocol
Logging: Category org.jgroups.protocols.dns with level TRACE
Initializing DNS Context with factory: com.sun.jndi.dns.DnsContextFactory and url
Resolving DNS query: eap-app-ping of a type: A <---------------- the query is sent to eap-app-ping service
Collect the entries: eap-app-3-gkcjb: entries collected from DNS (in 2 ms): [10.128.2.18:0, 10.128.2.17:0, 10.129.2.20:0, 10.131.0.34:0, 10.131.0.33:0] <-------------- collect
Send discovery request: Sending discovery request to 10.128.2.17:7600
Received: GET_MBRS_REQ and GET_MBRS_RSP for each member
Node will get GET_MBRS_RSP from other nodes, which will join the cluster: ISPN100000: Node eap-app-3-jvn9z joined the cluster
The reply will be name address and coordinator: sending response eap-app-3-gkcjb, name=eap-app-3-gkcjb, addr=10.128.2.18:7600, coord
Rebalance: ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z], phase READ_OLD_WRITE_ALL, topology id 2

GET_MBRS_REQ VS GET_MBRS_RSP

For the starting node the outgoing GET_MBRS_REQ (->) and incoming GET_MBRS_RSP (<-) are the important ones.

Interpreting

See INFO Configuring JGroups discovery protocol to dns.DNS_PING: mean discovery protocol == dns_ping

2022-12-22 18:08:10 Launching EAP Server
INFO Configuring JGroups cluster traffic encryption protocol to SYM_ENCRYPT.
[0;33mWARN Detected missing JGroups encryption configuration, the communication within the cluster WILL NOT be encrypted.[0m
INFO Configuring JGroups discovery protocol to dns.DNS_PING
INFO Using PicketBox SSL configuration.
INFO Found env LOGGER_CATEGORIES, configuring....
INFO Configuring logger category org.jgroups.protocols.dns with level TRACE
INFO Access log is disabled, ignoring configuration.
[0;33mWARN Configuration of an embedded messaging broker within the appserver is enabled but is not recommended. Support for such a configuration will be removed in a future release.[0m
[0;33mWARN If you are not configuring messaging destinations, to disable configuring an embedded messaging broker set the DISABLE_EMBEDDED_JMS_BROKER environment variable to true.[0m
INFO Server started in admin mode, CLI script executed during server boot.
INFO Running jboss-eap-7/eap74-openjdk8-runtime-openshift-rhel7 image, version 7.4.8
...
  JAVA_OPTS: -javaagent:"/opt/eap/jboss-modules.jar"  -server -verbose:gc -Xloggc:"/opt/eap/standalone/log/gc.log" -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms128m -Xmx512m -XX:MetaspaceSize=96m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=jdk.nashorn.api,com.sun.crypto.provider -Djava.awt.headless=true -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom  
....
[0m[0m18:08:12,433 INFO  [org.jboss.as] (MSC service thread 1-1) WFLYSRV0049: JBoss EAP 7.4.8.GA (WildFly Core 15.0.19.Final-redhat-00001) starting
	[Standalone] = 
	...
	jboss.bind.address = 10.128.2.18
	jboss.bind.address.management = 0.0.0.0
	jboss.bind.address.private = 10.128.2.18
	jboss.home.dir = /opt/eap
	jboss.host.name = eap-app-3-gkcjb
	jboss.messaging.cluster.password = <redacted>
	jboss.messaging.host = 10.128.2.18
	jboss.modules.dir = /opt/eap/modules
	jboss.modules.system.pkgs = jdk.nashorn.api,com.sun.crypto.provider
	jboss.node.name = eap-app-3-gkcjb
	jboss.qualified.host.name = eap-app-3-gkcjb
	jboss.server.base.dir = /opt/eap/standalone
	jboss.server.config.dir = /opt/eap/standalone/configuration
	jboss.server.data.dir = /opt/eap/standalone/data
	jboss.server.deploy.dir = /opt/eap/standalone/data/content
	jboss.server.log.dir = /opt/eap/standalone/log
	jboss.server.name = eap-app-3-gkcjb
	jboss.server.persist.config = true
	sun.java.command = /opt/eap/jboss-modules.jar -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -mp /opt/eap/modules org.jboss.as.standalone -Djboss.home.dir=/opt/eap -Djboss.server.base.dir=/opt/eap/standalone -c standalone-openshift.xml -bmanagement 0.0.0.0 -Djboss.server.data.dir=/opt/eap/standalone/data -Dwildfly.statistics-enabled=true -b 10.128.2.18 -bprivate 10.128.2.18 -Djboss.node.name=eap-app-3-gkcjb -Djboss.messaging.host=10.128.2.18 -Djboss.messaging.cluster.password=<redacted> --start-mode=admin-only -Dorg.wildfly.internal.cli.boot.hook.script=/tmp/cli-script-1671732490.cli -Dorg.wildfly.internal.cli.boot.hook.marker.dir=/tmp/cli-boot-reload-marker-1671732491 -Dorg.wildfly.internal.cli.boot.hook.script.properties=/tmp/cli-script-property-1671732490.cli -Dorg.wildfly.internal.cli.boot.hook.script.output.file=/tmp/cli-script-output-1671732490.cli -Dorg.wildfly.internal.cli.boot.hook.script.error.file=/tmp/cli-script-error-1671732490.cli -Dorg.wildfly.internal.cli.boot.hook.script.warn.file=/tmp/cli-warning-1671732490.log
	sun.java.launcher = SUN_STANDARD
	sun.jnu.encoding = ANSI_X3.4-1968
	sun.management.compiler = HotSpot 64-Bit Tiered Compilers
	sun.os.patch.level = unknown
	user.country = US
	user.dir = /home/jboss
	user.home = /home/jboss
	user.language = en
	user.name = 1000670000
	user.timezone = UTC
	wildfly.statistics-enabled = true
[0m[32m18:08:12,435 DEBUG [org.jboss.as.config] (MSC service thread 1-1) VM Arguments: -D[Standalone] -javaagent:/opt/eap/jboss-modules.jar -verbose:gc -Xloggc:/opt/eap/standalone/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms128m -Xmx512m -XX:MetaspaceSize=96m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=jdk.nashorn.api,com.sun.crypto.provider -Djava.awt.headless=true -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties 
...
[0m[0m18:08:13,977 INFO  [org.jboss.as.cli] (Controller Boot Thread) WFLYCLI0001: Processing CLI script /tmp/cli-script-1671732490.cli
[0m[0m18:08:14,434 INFO  [org.jboss.as.controller.management-deprecated] (CLI command executor) WFLYCTL0395: Operation add against the resource at address /subsystem=jgroups/stack=tcp/protocol=dns.DNS_PING/property=dns_query is deprecated, and it might be removed in future version. See the the output of the read-operation-description operation to learn more about the deprecation.
[0m[0m18:08:14,434 INFO  [org.jboss.as.controller.management-deprecated] (CLI command executor) WFLYCTL0395: Operation add against the resource at address /subsystem=jgroups/stack=tcp/protocol=dns.DNS_PING/property=async_discovery_use_separate_thread_per_request is deprecated, and it might be removed in future version. See the the output of the read-operation-description operation to learn more about the deprecation.
[0m[0m18:08:14,452 INFO  [org.jboss.as.controller.management-deprecated] (CLI command executor) WFLYCTL0395: Operation add against the resource at address /subsystem=jgroups/stack=udp/protocol=dns.DNS_PING/property=dns_query is deprecated, and it might be removed in future version. See the the output of the read-operation-description operation to learn more about the deprecation.
[0m[0m18:08:14,453 INFO  [org.jboss.as.controller.management-deprecated] (CLI command executor) WFLYCTL0395: Operation add against the resource at address /subsystem=jgroups/stack=udp/protocol=dns.DNS_PING/property=async_discovery_use_separate_thread_per_request is deprecated, and it might be removed in future version. See the the output of the read-operation-description operation to learn more about the deprecation.
...
[0m[32m18:08:14,511 DEBUG [org.jboss.as.config] (MSC service thread 1-2) Configured system properties:
	[Standalone] = 
	awt.toolkit = sun.awt.X11.XToolkit
	file.encoding = ANSI_X3.4-1968
	file.encoding.pkg = sun.io
	file.separator = /
	java.awt.graphicsenv = sun.awt.X11GraphicsEnvironment
	java.awt.headless = true
	java.awt.printerjob = sun.print.PSPrinterJob
	java.class.path = /opt/eap/jboss-modules.jar:/opt/eap/jboss-modules.jar
	java.class.version = 52.0
	java.endorsed.dirs = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre/lib/endorsed
	java.ext.dirs = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre/lib/ext:/usr/java/packages/lib/ext
	java.home = /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.352.b08-2.el7_9.x86_64/jre
	java.io.tmpdir = /tmp
	java.library.path = /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
	java.net.preferIPv4Stack = true
	java.runtime.name = OpenJDK Runtime Environment
	java.runtime.version = 1.8.0_352-b08
	java.security.egd = file:/dev/./urandom
	java.specification.maintenance.version = 4
	java.specification.name = Java Platform API Specification
	java.specification.vendor = Oracle Corporation
	java.specification.version = 1.8
	java.util.logging.manager = org.jboss.logmanager.LogManager
	java.vendor = Red Hat, Inc.
	java.vendor.url = https://www.redhat.com/
	java.vendor.url.bug = https://bugzilla.redhat.com/enter_bug.cgi?product=Red%20Hat%20Enterprise%20Linux%207&component=java-1.8.0-openjdk
	java.version = 1.8.0_352
	java.vm.info = mixed mode
	java.vm.name = OpenJDK 64-Bit Server VM
	java.vm.specification.name = Java Virtual Machine Specification
	java.vm.specification.vendor = Oracle Corporation
	java.vm.specification.version = 1.8
	java.vm.vendor = Red Hat, Inc.
	java.vm.version = 25.352-b08
	javax.management.builder.initial = org.jboss.as.jmx.PluggableMBeanServerBuilder
	jboss.bind.address = 10.128.2.18
	jboss.bind.address.management = 0.0.0.0
	jboss.bind.address.private = 10.128.2.18
	jboss.home.dir = /opt/eap
	jboss.host.name = eap-app-3-gkcjb
	jboss.messaging.cluster.password = <redacted>
	jboss.messaging.host = 10.128.2.18
	jboss.modules.dir = /opt/eap/modules
	jboss.modules.system.pkgs = jdk.nashorn.api,com.sun.crypto.provider
	jboss.node.name = eap-app-3-gkcjb
	jboss.qualified.host.name = eap-app-3-gkcjb
	jboss.server.base.dir = /opt/eap/standalone
	jboss.server.config.dir = /opt/eap/standalone/configuration
	jboss.server.data.dir = /opt/eap/standalone/data
	jboss.server.deploy.dir = /opt/eap/standalone/data/content
	jboss.server.log.dir = /opt/eap/standalone/log
	jboss.server.name = eap-app-3-gkcjb
	jboss.server.persist.config = true
	jboss.server.temp.dir = /opt/eap/standalone/tmp
	jolokia.agent = https://10.128.2.18:8778/jolokia/
	....
[0m[32m18:08:14,511 DEBUG [org.jboss.as.config] (MSC service thread 1-2) VM Arguments: -D[Standalone] -javaagent:/opt/eap/jboss-modules.jar -verbose:gc -Xloggc:/opt/eap/standalone/log/gc.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=5 -XX:GCLogFileSize=3M -XX:-TraceClassUnloading -Xms128m -Xmx512m -XX:MetaspaceSize=96m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=jdk.nashorn.api,com.sun.crypto.provider -Djava.awt.headless=true -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -Djava.security.egd=file:/dev/./urandom -Dorg.jboss.boot.log.file=/opt/eap/standalone/log/server.log -Dlogging.configuration=file:/opt/eap/standalone/configuration/logging.properties 
[0m[0m18:08:14,632 INFO  [org.jboss.as.repository] (ServerService Thread Pool -- 9) WFLYDR0001: Content added at location /opt/eap/standalone/data/content/3f/99ba57ad67304be65f7f751400949523f75236/content
...
[0m[0m18:08:14,833 INFO  [org.jboss.as.naming] (MSC service thread 1-1) WFLYNAM0003: Starting Naming Service
[0m[0m18:08:14,913 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0012: Started server default-server.
[0m[0m18:08:14,991 INFO  [org.jboss.as.ejb3] (MSC service thread 1-1) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 4 (per class), which is derived from the number of CPUs on this host.
[0m[0m18:08:14,991 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) Queuing requests.
[0m[0m18:08:14,992 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0018: Host default-host starting
[0m[0m18:08:14,996 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0006: Undertow HTTP listener default listening on 0.0.0.0:8080
[0m[0m18:08:15,032 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "ROOT.war" (runtime-name: "ROOT.war")
[0m[0m18:08:15,034 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/eap/standalone/deployments
[0m[0m18:08:15,038 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0493: Jakarta Enterprise Beans subsystem suspension complete
[0m[0m18:08:15,108 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 62) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=/opt/eap/standalone/data/activemq/journal,bindingsDirectory=/opt/eap/standalone/data/activemq/bindings,largeMessagesDirectory=/opt/eap/standalone/data/activemq/largemessages,pagingDirectory=/opt/eap/standalone/data/activemq/paging)
....
[0m[0m18:08:15,472 INFO  [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0003: Processing weld deployment ROOT.war
[0m[0m18:08:15,529 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-1) WFLYJCA0007: Registered connection factory java:/JmsXA
[0m[0m18:08:15,582 INFO  [org.jboss.as.connector.deployment] (MSC service thread 1-1) WFLYJCA0002: Bound Jakarta Connectors ConnectionFactory [java:/JmsXA]
[0m[0m18:08:16,398 INFO  [org.infinispan.CONTAINER] (ServerService Thread Pool -- 64) ISPN000128: Infinispan version: Infinispan 'Corona Extra' 11.0.16.Final-redhat-00001
[0m[0m18:08:16,450 INFO  [org.infinispan.CONFIG] (MSC service thread 1-1) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
[0m[0m18:08:16,527 INFO  [org.infinispan.PERSISTENCE] (ServerService Thread Pool -- 64) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.spi.marshalling.InfinispanProtoStreamMarshaller'
[0m[0m18:08:16,749 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 64) WFLYCLINF0002: Started http-remoting-connector cache from ejb container
[0m[0m18:08:16,955 INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: JBoss EAP 7.4.8.GA (WildFly Core 15.0.19.Final-redhat-00001) started in 2444ms - Started 417 of 709 services (437 services are lazy, passive or on-demand)

[0m[0m18:11:07,102 INFO  [org.jboss.as.repository] (DeploymentScanner-threads - 1) WFLYDR0001: Content added at location /opt/eap/standalone/data/content/33/d9d526f12c45adf6bbd0372e1f240fdd7026db/content
[0m[0m18:11:07,105 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "clusterbench-ee8.ear" (runtime-name: "clusterbench-ee8.ear")
[0m[0m18:11:07,134 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0207: Starting subdeployment (runtime-name: "clusterbench-ee8-web-passivating.war")
...
[0m[33m18:11:07,327 WARN  [org.jboss.as.dependency.unsupported] (MSC service thread 1-1) WFLYSRV0019: Deployment "deployment.clusterbench-ee8.ear.clusterbench-ee8-web-granular.war" is using an unsupported module ("org.jgroups") which may be changed or removed in future versions without notice.
[0m[33m18:11:07,331 WARN  [org.jboss.as.dependency.unsupported] (MSC service thread 1-2) WFLYSRV0019: Deployment "deployment.clusterbench-ee8.ear.clusterbench-ee8-web.war" is using an unsupported module ("org.jgroups") which may be changed or removed in future versions without notice.
[0m[33m18:11:07,333 WARN  [org.jboss.as.dependency.unsupported] (MSC service thread 1-1) WFLYSRV0019: Deployment "deployment.clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war" is using an unsupported module ("org.jgroups") which may be changed or removed in future versions without notice.
[0m[0m18:11:07,334 INFO  [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0003: Processing weld deployment clusterbench-ee8.ear
[0m[0m18:11:07,372 INFO  [org.jboss.weld.deployer] (MSC service thread 1-2) WFLYWELD0003: Processing weld deployment clusterbench-ee8-web-passivating.war
[0m[0m18:11:07,374 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment clusterbench-ee8-ejb.jar
[0m[0m18:11:07,383 INFO  [org.jboss.as.ejb3.deployment] (MSC service thread 1-1) WFLYEJB0473: JNDI bindings for session bean named 'RemoteStatelessSBImpl' in deployment unit 'subdeployment "clusterbench-ee8-ejb.jar" of deployment "clusterbench-ee8.ear"' are as follows:
...
	java:global/clusterbench-ee8/clusterbench-ee8-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
	java:app/clusterbench-ee8-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
	java:module/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
	java:jboss/exported/clusterbench-ee8/clusterbench-ee8-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
	ejb:clusterbench-ee8/clusterbench-ee8-ejb/RemoteStatelessSBImpl!org.jboss.test.clusterbench.ejb.stateless.RemoteStatelessSB
	java:global/clusterbench-ee8/clusterbench-ee8-ejb/RemoteStatelessSBImpl
	java:app/clusterbench-ee8-ejb/RemoteStatelessSBImpl
	java:module/RemoteStatelessSBImpl
....
[0m[0m18:11:07,391 INFO  [org.jboss.weld.deployer] (MSC service thread 1-1) WFLYWELD0003: Processing weld deployment clusterbench-ee8-web.war
[0m[33m18:11:07,472 WARN  [org.wildfly.clustering.web.undertow] (MSC service thread 1-1) WFLYCLWEBUT0004: Legacy <replication-config/> overriding attached distributable session management provider for clusterbench-ee8-web-granular.war
[0m18:11:07,609 TRACE [org.jgroups.protocols.dns.DefaultDNSResolver] (ServerService Thread Pool -- 9) initializing DNS Context with factory: com.sun.jndi.dns.DnsContextFactory and url: 
[0m18:11:07,687 TRACE [org.jgroups.protocols.dns.DefaultDNSResolver] (ServerService Thread Pool -- 9) resolving DNS query: eap-app-ping of a type: A <---------------------------------------------- the query is sent to eap-app-ping
[0m[32m18:11:07,689 DEBUG [org.jgroups.protocols.dns.DNS_PING] (ServerService Thread Pool -- 9) eap-app-3-gkcjb: entries collected from DNS (in 2 ms): [10.128.2.18:0, 10.128.2.17:0, 10.129.2.20:0, 10.131.0.34:0, 10.131.0.33:0] <-------------- collect
[0m[32m18:11:07,689 DEBUG [org.jgroups.protocols.dns.DNS_PING] (ServerService Thread Pool -- 9) eap-app-3-gkcjb: sending discovery requests to hosts [10.128.2.18:0, 10.128.2.17:0, 10.129.2.20:0, 10.131.0.34:0, 10.131.0.33:0] on ports [7600 .. 7600] <-----
[0m18:11:07,692 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-4,null,eap-app-3-gkcjb) eap-app-3-gkcjb: sending discovery request to 10.128.2.17:7600
[0m18:11:07,692 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-7,ee,eap-app-3-gkcjb) eap-app-3-gkcjb: sending discovery request to 10.129.2.20:7600
[0m18:11:07,692 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-5,null,eap-app-3-gkcjb) eap-app-3-gkcjb: sending discovery request to 10.128.2.18:7600
[0m18:11:07,692 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-8,ee,eap-app-3-gkcjb) eap-app-3-gkcjb: sending discovery request to 10.131.0.34:7600
[0m18:11:07,694 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-9,ee,eap-app-3-gkcjb) eap-app-3-gkcjb: sending discovery request to 10.131.0.33:7600
[0m18:11:07,696 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-10,ee,eap-app-3-gkcjb) eap-app-3-gkcjb: received GET_MBRS_REQ from eap-app-3-gkcjb, sending response eap-app-3-gkcjb, name=eap-app-3-gkcjb, addr=10.128.2.18:7600
[0m18:11:07,697 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-5,null,eap-app-3-gkcjb) eap-app-3-gkcjb: received GET_MBRS_RSP from eap-app-3-gkcjb: eap-app-3-gkcjb, name=eap-app-3-gkcjb, addr=10.128.2.18:7600
[0m[0m18:11:10,759 INFO  [org.infinispan.PERSISTENCE] (ServerService Thread Pool -- 9) ISPN000556: Starting user marshaller 'org.wildfly.clustering.infinispan.spi.marshalling.InfinispanProtoStreamMarshaller'
[0m[0m18:11:10,763 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 9) ISPN000078: Starting JGroups channel ee
[0m[0m18:11:10,765 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 9) ISPN000094: Received new cluster view for channel ee: [eap-app-3-gkcjb|0] (1) [eap-app-3-gkcjb]
[0m[0m18:11:10,769 INFO  [org.infinispan.CLUSTER] (ServerService Thread Pool -- 9) ISPN000079: Channel ee local address is eap-app-3-gkcjb, physical addresses are [10.128.2.18:7600]
[0m[0m18:11:10,777 INFO  [org.infinispan.CONFIG] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
[0m[0m18:11:10,777 INFO  [org.infinispan.CONFIG] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
[0m[0m18:11:10,778 INFO  [org.infinispan.CONFIG] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
[0m[0m18:11:10,779 INFO  [org.infinispan.CONFIG] (MSC service thread 1-2) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
[0m[33m18:11:10,816 WARN  [org.jboss.weld.Bootstrap] (MSC service thread 1-2) WELD-000146: BeforeBeanDiscovery.addAnnotatedType(AnnotatedType<?>) used for class com.sun.faces.flow.FlowDiscoveryCDIHelper is deprecated from CDI 1.1!
[0m[0m18:11:11,039 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 9) WFLYCLINF0002: Started default-server cache from web container
[0m[0m18:11:11,064 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 70) WFLYCLINF0002: Started clusterbench-ee8.ear.clusterbench-ee8-web.war cache from web container
[0m[0m18:11:11,090 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 69) WFLYCLINF0002: Started clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war cache from web container
[0m[0m18:11:11,091 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 68) WFLYCLINF0002: Started clusterbench-ee8.ear.clusterbench-ee8-web-granular.war cache from web container
[0m[31m18:11:11,098 SEVERE [javax.enterprise.resource.webcontainer.jsf.application.view] (MSC service thread 1-2) Unable to obtain CDI 1.1 utilities for Mojarra
[0m[31m18:11:11,100 SEVERE [javax.enterprise.resource.webcontainer.jsf.flow] (MSC service thread 1-2) Unable to obtain CDI 1.1 utilities for Mojarra
[0m[0m18:11:11,230 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 9) WFLYUT0021: Registered web context: '/clusterbench-granular' for server 'default-server'
[0m[0m18:11:11,248 INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 74) Initializing Mojarra 2.3.14.SP06 for context '/clusterbench-passivating'
[0m[0m18:11:11,250 INFO  [javax.enterprise.resource.webcontainer.jsf.config] (ServerService Thread Pool -- 75) Initializing Mojarra 2.3.14.SP06 for context '/clusterbench'
[0m[0m18:11:11,595 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 75) WFLYUT0021: Registered web context: '/clusterbench' for server 'default-server'
[0m[0m18:11:11,599 INFO  [org.wildfly.extension.undertow] (ServerService Thread Pool -- 74) WFLYUT0021: Registered web context: '/clusterbench-passivating' for server 'default-server'
[0m[0m18:11:11,613 INFO  [org.jboss.as.server] (DeploymentScanner-threads - 1) WFLYSRV0010: Deployed "clusterbench-ee8.ear" (runtime-name : "clusterbench-ee8.ear")
[0m18:11:15,001 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-9,ee,eap-app-3-gkcjb) eap-app-3-gkcjb: received GET_MBRS_REQ from eap-app-3-jvn9z, sending response eap-app-3-gkcjb, name=eap-app-3-gkcjb, addr=10.128.2.18:7600, coord
[0m[0m18:11:15,015 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) ISPN000094: Received new cluster view for channel ee: [eap-app-3-gkcjb|1] (2) [eap-app-3-gkcjb, eap-app-3-jvn9z]
[0m[0m18:11:15,017 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) ISPN100000: Node eap-app-3-jvn9z joined the cluster
[0m[0m18:11:15,306 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z], phase READ_OLD_WRITE_ALL, topology id 2
[0m[0m18:11:15,432 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z], phase READ_OLD_WRITE_ALL, topology id 2
[0m[0m18:11:15,435 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z], phase READ_OLD_WRITE_ALL, topology id 2
[0m[0m18:11:15,436 INFO  [org.infinispan.CLUSTER] (thread-4,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z], phase READ_OLD_WRITE_ALL, topology id 2
[0m[0m18:11:15,486 INFO  [org.infinispan.CLUSTER] (thread-4,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3
[0m[0m18:11:15,487 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3
[0m[0m18:11:15,498 INFO  [org.infinispan.CLUSTER] (thread-4,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4
[0m[0m18:11:15,505 INFO  [org.infinispan.CLUSTER] (thread-4,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3
[0m[0m18:11:15,511 INFO  [org.infinispan.CLUSTER] (thread-4,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4
[0m[0m18:11:15,511 INFO  [org.infinispan.CLUSTER] (thread-7,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z], topology id 5
[0m[0m18:11:15,512 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4
[0m[0m18:11:15,528 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 3
[0m[0m18:11:15,546 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p7-t2) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z], topology id 5
[0m[0m18:11:15,546 INFO  [org.infinispan.CLUSTER] (non-blocking-thread--p7-t1) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z], topology id 5
[0m[0m18:11:15,560 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 4
[0m[0m18:11:15,566 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z], topology id 5
[0m18:11:18,790 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-9,ee,eap-app-3-gkcjb) eap-app-3-gkcjb: received GET_MBRS_REQ from eap-app-3-msczt, sending response eap-app-3-gkcjb, name=eap-app-3-gkcjb, addr=10.128.2.18:7600, coord
[0m[0m18:11:18,797 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) ISPN000094: Received new cluster view for channel ee: [eap-app-3-gkcjb|2] (3) [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt]
[0m[0m18:11:18,797 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) ISPN100000: Node eap-app-3-msczt joined the cluster
[0m[0m18:11:19,301 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt], phase READ_OLD_WRITE_ALL, topology id 6
[0m[0m18:11:19,428 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt], phase READ_OLD_WRITE_ALL, topology id 6
[0m[0m18:11:19,437 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt], phase READ_OLD_WRITE_ALL, topology id 6
[0m[0m18:11:19,441 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt], phase READ_OLD_WRITE_ALL, topology id 6
[0m[0m18:11:19,492 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7
[0m[0m18:11:19,493 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7
[0m[0m18:11:19,494 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7
[0m[0m18:11:19,508 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8
[0m[0m18:11:19,509 INFO  [org.infinispan.CLUSTER] (thread-7,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8
[0m[0m18:11:19,509 INFO  [org.infinispan.CLUSTER] (thread-4,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8
[0m[0m18:11:19,510 INFO  [org.infinispan.CLUSTER] (thread-7,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 7
[0m[0m18:11:19,513 INFO  [org.infinispan.CLUSTER] (thread-4,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt], topology id 9
[0m[0m18:11:19,525 INFO  [org.infinispan.CLUSTER] (thread-7,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt], topology id 9
[0m[0m18:11:19,525 INFO  [org.infinispan.CLUSTER] (thread-5,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt], topology id 9
[0m[0m18:11:19,527 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 8
[0m[0m18:11:19,546 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt], topology id 9
[0m18:11:22,751 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-8,ee,eap-app-3-gkcjb) eap-app-3-gkcjb: received GET_MBRS_REQ from eap-app-3-qthbb, sending response eap-app-3-gkcjb, name=eap-app-3-gkcjb, addr=10.128.2.18:7600, coord
[0m[0m18:11:22,756 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) ISPN000094: Received new cluster view for channel ee: [eap-app-3-gkcjb|3] (4) [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb]
[0m[0m18:11:22,756 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) ISPN100000: Node eap-app-3-qthbb joined the cluster
[0m[0m18:11:23,037 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb], phase READ_OLD_WRITE_ALL, topology id 10
[0m[0m18:11:23,201 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb], phase READ_OLD_WRITE_ALL, topology id 10
[0m[0m18:11:23,203 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb], phase READ_OLD_WRITE_ALL, topology id 10
[0m[0m18:11:23,204 INFO  [org.infinispan.CLUSTER] (thread-5,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb], phase READ_OLD_WRITE_ALL, topology id 10
[0m[0m18:11:23,249 INFO  [org.infinispan.CLUSTER] (thread-4,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 11
[0m[0m18:11:23,250 INFO  [org.infinispan.CLUSTER] (thread-5,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 11
[0m[0m18:11:23,250 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 11
[0m[0m18:11:23,257 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 12
[0m[0m18:11:23,262 INFO  [org.infinispan.CLUSTER] (thread-5,null,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 12
[0m[0m18:11:23,266 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 12
[0m[0m18:11:23,271 INFO  [org.infinispan.CLUSTER] (thread-10,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 11
[0m[0m18:11:23,276 INFO  [org.infinispan.CLUSTER] (thread-7,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb], topology id 13
[0m[0m18:11:23,281 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb], topology id 13
[0m[0m18:11:23,282 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb], topology id 13
[0m[0m18:11:23,292 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 12
[0m[0m18:11:23,300 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb], topology id 13
[0m18:11:27,806 TRACE [org.jgroups.protocols.dns.DNS_PING] (thread-8,ee,eap-app-3-gkcjb) eap-app-3-gkcjb: received GET_MBRS_REQ from eap-app-3-qzd4z, sending response eap-app-3-gkcjb, name=eap-app-3-gkcjb, addr=10.128.2.18:7600, coord
[0m[0m18:11:27,810 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) ISPN000094: Received new cluster view for channel ee: [eap-app-3-gkcjb|4] (5) [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb, eap-app-3-qzd4z]
[0m[0m18:11:27,810 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) ISPN100000: Node eap-app-3-qzd4z joined the cluster
[0m[0m18:11:28,099 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb, eap-app-3-qzd4z], phase READ_OLD_WRITE_ALL, topology id 14
[0m[0m18:11:28,251 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb, eap-app-3-qzd4z], phase READ_OLD_WRITE_ALL, topology id 14
[0m[0m18:11:28,255 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb, eap-app-3-qzd4z], phase READ_OLD_WRITE_ALL, topology id 14
[0m[0m18:11:28,263 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100002: Starting rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb, eap-app-3-qzd4z], phase READ_OLD_WRITE_ALL, topology id 14
[0m[0m18:11:28,335 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 15
[0m[0m18:11:28,335 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 15
[0m[0m18:11:28,336 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 15
[0m[0m18:11:28,339 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_ALL_WRITE_ALL, topology id 15
[0m[0m18:11:28,342 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 16
[0m[0m18:11:28,343 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 16
[0m[0m18:11:28,352 INFO  [org.infinispan.CLUSTER] (thread-8,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 16
[0m[0m18:11:28,352 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-granular.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb, eap-app-3-qzd4z], topology id 17
[0m[0m18:11:28,352 INFO  [org.infinispan.CLUSTER] (thread-9,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100009: Advancing to rebalance phase READ_NEW_WRITE_ALL, topology id 16
[0m[0m18:11:28,370 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web-passivating.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb, eap-app-3-qzd4z], topology id 17
[0m[0m18:11:28,374 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=default-server] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb, eap-app-3-qzd4z], topology id 17
[0m[0m18:11:28,384 INFO  [org.infinispan.CLUSTER] (thread-18,ee,eap-app-3-gkcjb) [Context=clusterbench-ee8.ear.clusterbench-ee8-web.war] ISPN100010: Finished rebalance with members [eap-app-3-gkcjb, eap-app-3-jvn9z, eap-app-3-msczt, eap-app-3-qthbb, eap-app-3-qzd4z], topology id 17
[0m

Root Cause

The solution EAP 7 image clustering in OCP 4 bring several aspects of clustering in EAP 7.

Avoid restricting DNS namespace comms.

Avoid restricting DNS namespace comms, as detailed on Pods are having issues with DNS after the network policies are applied. Remember to always allow openshift-dns namespace communication.

Issue	Solution
EAP 7 clustering (main solution)	EAP 7 image clustering in OCP 4
Deployment boundaries: labels of service	EAP 7 clustering deployments boundaries in OCP cluster
Istio/Service Mesh with EAP 7	EAP 7 deployment with Service Mesh
To set custom configurations	What are the options to use a custom runtime JBoss EAP 7 configurations in OCP

Diagnostic Steps

Setting org.jgroups.protocols.DNS_PING with level TRACE:

# set DNS_PING
$ oc set env dc/eap-app LOGGER_CATEGORIES=org.jgroups.protocols.DNS_PING:TRACE
...
# confirming DNS_PING to TRACE
INFO Found env LOGGER_CATEGORIES, configuring....
INFO Configuring logger category org.jgroups.protocols.DNS_PING with level TRACE <----

For jgroups.protocols set org.jgroups:TRACE:

oc set env dc/eap-app LOGGER_CATEGORIES=org.jgroups.protocols.DNS_PING:TRACE,org.jgroups:TRACE

Search for GET_MBRS_REQ and GET_MBRS_RSP - for outgoing and incoming nodes;
EAP 7's service will be eap-app-ping by default and it will use deploymentConfig label by default
The EAP 7's default service is headless, so one cannot curl the service.
oc get ep,svc -o wide to get the services and the endpoints:

### service
oc get svc -o wide
server/eap-app-ping Type ClusterIP None <none> 8888 75M  selector: deploymentConfig=eap-app
### endpoint
oc get ep -o wide
NAME                                                     ENDPOINTS                                                          AGE
endpoints/eap-app-ping                                   1.1.1.127:8888, 1.1.1.128:8888                3 min

SBR

JBoss Base AS

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Learn more

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.

EAP 7 DNS_PING tracing logs in OCP 4

Environment

Issue

Resolution

For enabling DNS_PING tracing logging

Process

GET_MBRS_REQ VS GET_MBRS_RSP

Interpreting

Root Cause

Avoid restricting DNS namespace comms.

For enabling DNS_PING tracing logging

Clustering deployment boundaries

EAP 7 deployment with Service Mesh

Diagnostic Steps