During startup of JVM(s) 'WFLYCTL0216: Management resource '[("subsystem" => "datasources")]' not found' observed in JBoss EAP when Dynatrace is enabled

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7
• Red Hat Single Sign-On (RH-SSO)
  • 7
• Openshift

Issue

• Observing errors like the following during JVM startup

... ERROR [org.jboss.as.controller.management-operation] (oneagentperiodiceventsmanaged) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
    ("subsystem" => "datasources"),
    ("data-source" => "MYDS")
]) - failure description: "WFLYCTL0216: Management resource '[(\"subsystem\" => \"datasources\")]' not found"

• We are using Dynatrace monitoring with JBoss EAP and seeing errors like this before JBoss EAP has started:

... ERROR [org.jboss.as.controller.management-operation] (oneagentperiodiceventsmanaged) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
    ("subsystem" => "batch-jberet"),
    ("thread-pool" => "batch")
]) - failure description: "WFLYCTL0216: Management resource '[(\"subsystem\" => \"batch-jberet\")]' not found"
...
ERROR [org.jboss.as.controller.management-operation] (oneagentperiodiceventsmanaged) WFLYCTL0013: Operation ("read-attribute") failed - address: ([
    ("subsystem" => "ejb3"),
    ("thread-pool" => "default")

Resolution

Dynatrace not in OpenShift

Dynatrace is trying to call services before EAP is started

• The ERRORs can be ignored if they are not causing a problem beyond the log messages
• Contact Dynatrace Support for further assistance. It seems the Dynatrace Support is aware of this issue, indicating that the early start of the monitoring for a JBoss application is a well known issue and that one can delay the start for one jvm server instance (or pod) with the environment variable DT_DEBUGFLAGS="optionDelayJmxMetricsMinJava=300".
  This environment variable needs to be set at OS level ( Ex., For RHEL, one can set it in .bash_profile)

Dynatrace in OpenShift

The same as above when not in OpenShift except:

• There was a bug in EAP where the ERROR logged when Dyntrace tries to call the management interface to check a subsystem before EAP is started was being treated as a boot error, instead of an error during boot (caused by an application calling a service which was not yet available), which caused OpenShift to report the EAP Container readiness probe (/opt/eap/bin/readinessProbe.sh) as having issues, when the boot was fine. This bug was fixed to distinguish between the boot error and error during boot, Update to EAP 7.3 Update 8+ (or RH-SSO 7.4 Update 8+ if using RH-SSO) as per bug fix This content is not included.JBEAP-21845. The same issue is also being addressed in EAP 7.4 Update 1+ as per bug fix This content is not included.JBEAP-21839.
• When running EAP (or RH-SSO) as Openshift container image, it is also possible to workaround this issue by setting an environment variable as PROBE_DISABLE_BOOT_ERRORS_CHECK with a value of true (see Content from github.com is not included.Content from github.com is not included.https://github.com/jboss-openshift/cct_module/blob/sprint-18/os-eap-probes/added/probes/probe/eap/dmr.py#L78). This will disable the execution of the BootErrorsTest (aka, test probe.eap.dmr.BootErrorsTest ).

Notes

• The errors, when observed during JVM startup, may indicate that the monitoring agent is running too early though the datasource pools may be successfully deployed upon completion of JVM startup.
• Any ERROR message where the thread oneagentperiodiceventsmanaged is trying to invoke a management operation before the JBoss EAP ... started message is logged indicate the same Dynatrace issue trying to invoke management operations before EAP is started.

INFO  [org.jboss.as] (Controller Boot Thread) WFLYSRV0025: JBoss EAP 7.4.6.GA (WildFly Core 15.0.15.Final-redhat-00001) started

• The same kind issues are reported with NewRelic in Content from github.com is not included.JBoss Wildfly throws error on startup #229. This is also because the NewRelic agent executes Management Operations while JBoss EAP is still booting.

Root Cause

• The oneagentperiodiceventsmanaged thread is associated with Dynatrace "One Agent" monitoring which may be running too early during JVM startup.
• Earlier EAP 7 versions incorrectly consider any errors during startup as a boot error, which OpenShift readiness script considers a failure.
• See also: This content is not included.WFCORE-5368: Populating the boot error collector does not distinguish between problems that happen as part of boot vs those that happen during boot

Diagnostic Steps

• Verify that the datasources are successfully deployed/working after startup completes

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.