JBoss EAP fails to initialize with log message "WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager" when an agent uses JUL

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 8.x
    • 7.x
    • 6.x
  • JDK
    • 11
    • 8
    • 7
    • 6

Issue

  • Using one of the following system properties, and I get the IllegalStateException below:
-Dcom.sun.management.jmxremote=x
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
  • Getting below logManager error on startup of EAP 6.
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
Oct 14, 2014 5:04:56 PM org.jboss.msc.service.ServiceContainerImpl <clinit>
INFO: JBoss MSC version 1.1.5.Final-redhat-1
Oct 14, 2014 5:04:56 PM org.jboss.as.server.ApplicationServerService start
INFO: JBAS015899: JBoss EAP 6.3.1.GA (AS 7.4.1.Final-redhat-3) starting
Oct 14, 2014 5:04:58 PM org.jboss.as.controller.AbstractOperationContext executeStep
ERROR: JBAS014612: Operation ("parallel-extension-add") failed - address: ([])
java.lang.RuntimeException: JBAS014670: Failed initializing module org.jboss.as.logging
	at org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:111)
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:611)
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:489)
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:290)
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:285)
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1132)
	at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:299)
	at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:292)
	at org.jboss.as.server.ServerService.boot(ServerService.java:346)
	at org.jboss.as.server.ServerService.boot(ServerService.java:321)
	at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:254)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: JBAS011592: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:188)
	at org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:103)
	... 11 more
Caused by: java.lang.IllegalStateException: JBAS011592: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
	at org.jboss.as.logging.LoggingExtension.initialize(LoggingExtension.java:106)
	at org.jboss.as.controller.extension.ExtensionAddHandler.initializeExtension(ExtensionAddHandler.java:97)
	at org.jboss.as.controller.extension.ParallelExtensionAddHandler$ExtensionInitializeTask.call(ParallelExtensionAddHandler.java:139)
	at org.jboss.as.controller.extension.ParallelExtensionAddHandler$ExtensionInitializeTask.call(ParallelExtensionAddHandler.java:125)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Oct 14, 2014 5:04:58 PM org.jboss.as.server.ServerService boot
FATAL: JBAS015957: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details.
  • Installed a Java agent (e.g. Wily Introscope) via the -javaagent JVM command-line parameter. When trying to boot JBoss, getting the following error stack trace:
java.lang.IllegalStateException: The LogManager was not properly installed (you must set the "java.util.logging.manager" system property to "org.jboss.logmanager.LogManager")
	at org.jboss.logmanager.Logger.getLogger(Logger.java:61)
	at org.jboss.as.server.Main.main(Main.java:84)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.jboss.modules.Module.run(Module.java:270)
	at org.jboss.modules.Main.main(Main.java:294)

For reference , here is a very basic Java agent implementation Content from bit.ly is not included.Content from bit.ly is not included.http://bit.ly/UjXkN6 where the issue is easily reproducible

  • JBoss fails to start after numerous attempts. The following message is captured in the logs when startup fails
WARNING: Failed to load the specified log manager class org.jboss.logmanager.LogManager
Oct 14, 2014 4:53:04 PM org.jboss.msc.service.ServiceContainerImpl <clinit>
INFO: JBoss MSC version 1.1.5.Final-redhat-1
Oct 14, 2014 4:53:05 PM org.jboss.as.server.ApplicationServerService start
INFO: JBAS015899: JBoss EAP 6.3.0.GA (AS 7.4.0.Final-redhat-19) starting
Oct 14, 2014 4:53:29 PM org.jboss.as.controller.AbstractOperationContext executeStep
ERROR: JBAS014612: Operation ("parallel-extension-add") failed - address: ([])
java.lang.RuntimeException: JBAS014670: Failed initializing module org.jboss.as.logging
	at org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:111)
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:611)
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:489)
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:290)
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:285)
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1132)
	at org.jboss.as.controller.ModelControllerImpl.boot(ModelControllerImpl.java:299)
	at org.jboss.as.controller.AbstractControllerService.boot(AbstractControllerService.java:292)
	at org.jboss.as.server.ServerService.boot(ServerService.java:346)
	at org.jboss.as.server.ServerService.boot(ServerService.java:321)
	at org.jboss.as.controller.AbstractControllerService$1.run(AbstractControllerService.java:254)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalStateException: JBAS011592: The logging subsystem requires the log manager to be org.jboss.logmanager.LogManager. The subsystem has not be initialized and cannot be used. To use JBoss Log Manager you must add the system property "java.util.logging.manager" and set it to "org.jboss.logmanager.LogManager"
	at java.util.concurrent.FutureTask.report(FutureTask.java:122)
	at java.util.concurrent.FutureTask.get(FutureTask.java:188)
	at org.jboss.as.controller.extension.ParallelExtensionAddHandler$1.execute(ParallelExtensionAddHandler.java:103)
	... 11 more
.. .. ..
.. .. ..

Resolution

EAP 7.4.x or 8.x

Remove the -javaagent from JAVA_OPTS and configure it in MODULE_OPTS instead in standalone.conf (or standalone.conf.bat/standalone.conf.ps1 for Windows).

MODULE_OPTS="-javaagent:/path/to/myagent.jar"

JDK 8/7

Oracle (SUN) JMX Agent


With regards to supportability, the **Sun JMX agent is not supported by Red Hat, and we recommend you connect to the JMX server using the JBoss `jmx-remoting` protocol instead.** Please review solution [How to Access MBeanServerConnection from remote JMX Client in JBoss EAP 6](https://access.redhat.com/site/solutions/125473) configuration details.

If Sun JMX Agent is enabled in JBoss EAP 7.4, MBean's ObjectName jboss.as:* will not be exposed. It means we can not see key metrics of JBoss EAP such as datasource pool metrics, transaction metrics, and io subsystem's busyTaskThreadCount. Hence we do not recommend enabling the Sun JMX Agent in JBoss EAP.

Third party Java Agents


Also note that third party Java agents (e.g. Wily/Introscope) need to ensure they allow JBoss to properly initialize the `java.util.logging.LogManager`. The following information will help you configure your JBoss installation to work around these issues, but these instructions are not fully supported by Red Hat, meaning cases regarding problems with following the instructions in this document will be worked on a *commercially reasonable basis*.
Configuration changes for Linux

In your `standalone.conf` or `domain.conf`:

  1. Add -Djava.util.logging.manager=org.jboss.logmanager.LogManager to JAVA_OPTS. For example, add the following to the last line:

       JAVA_OPTS="$JAVA_OPTS -Djava.util.logging.manager=org.jboss.logmanager.LogManager"

  2. Add -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-<version>.jar to JAVA_OPTS (make sure the JAR name and path matches your installation)

       JAVA_OPTS="$JAVA_OPTS -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-<version>.jar"

  3. Add org.jboss.logmanager to JBOSS_MODULES_SYSTEM_PKGS="..." defined in line 38. For example:

       37 if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
   38    JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,org.jboss.logmanager"
   39 fi
Configuration changes for Windows

In your `standalone.conf.bat` / `domain.conf.bat` would be:**

  1. Add this line at the end of the file, after the :JAVA_OPTS_SET :

       set "JAVA_OPTS=%JAVA_OPTS% -Djava.util.logging.manager=org.jboss.logmanager.LogManager"

  2. Add this line at the end of the file, after the 'JAVA_OPTS_SET : (make sure the JAR name and path matches your installation)

       set "JAVA_OPTS=%JAVA_OPTS% -Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-<version>.jar"

  3. Change this line from: 

       set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.byteman"

    to: 

       set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.byteman,org.jboss.logmanager"
Configuration changes for Domain mode

In the domain.xml add `-Xbootclasspath/p:$JBOSS_HOME/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-.jar` to `jvm-options` under server-group (make sure the server group is using the agent) 
          1 <server-groups>
          2   <server-group name="server-group" profile="full">
          3    <jvm name="default">
          4       <heap size="128m" max-size="128m"/>
          5       <permgen size="64m" max-size="128m"/>
          6       <jvm-options>
          7           <option value="-Xbootclasspath/p:${env.JBOSS_HOME}/modules/system/layers/base/org/jboss/logmanager/main/jboss-logmanager-<version>.jar"/>
          8       </jvm-options>
          9   </jvm>

This will:

  • Add JBoss LogManager to the boot classloader so it will be available to the Java agent when it initializes logging.
  • Ensure java.util.logging.LogManager is configured to use JBoss LogManager as its LogManager implementation

An alternative to step 3 is to modify the module.xml for the module org.jboss.logmanager like so:

<module xmlns="urn:jboss:module:1.1" name="org.jboss.logmanager">
  <dependencies>
    <system export="true">
      <paths>
        <path name="org/jboss/logmanager"/>
        <path name="org/jboss/logmanager/config"/>
        <path name="org/jboss/logmanager/errormanager"/>
        <path name="org/jboss/logmanager/filters"/>
        <path name="org/jboss/logmanager/formatters"/>
        <path name="org/jboss/logmanager/handlers"/>
      </paths>
    </system>
  </dependencies>
</module>

This alternative solution will cause JBoss Modules to use the copy of the classes from the boot classloader instead of creating its own copy when it initializes. It also prevents the JBoss LogManager classes from being visible to all deployments and modules.

Note: There is Content from issues.jboss.org is not included.a bug identified in EAP 6.1.0 that causes this workaround to fail. The bug is fixed in the cumulative patch release EAP 6.1.1 or you can see this knowledgebase solution for a workaround.

Note: JBoss EAP 6 disables old jars when a cumulative patch is applied for security reasons, due to this, after applying a cumulative patch you need to update the -Xbootclasspath ... .../jboss-logmanager-.jar to point to the updated jar. Why are some JARs unusable after applying a patch in JBoss EAP 6

JDK 11

Xbootclasspath/p is removed from OpenJDK 9 and instead, Xbootclasspath/a can be used as a workaround. However, this workaround does not work for JDK 11 due to the reason that the current JAR is an MR JAR with the base being Java 8. In the Java portion, there is an org.jboss.logmanager.JBossLoggerProvider and a META-INF/services/java.lang.System$LoggerProvider. In normal situations this works fine, however, the boot classpath does not support MR JAR.

JBoss EAP 7.4

In JBoss EAP 7.4, to simplify the process above, the MODULE_OPTS variable was introduced where one can set a java agent. The varible MODULE_OPTS can be used for JDK 8 but also JDK 11.

For Linux platform, edit standalone.conf, located at $JBOSS_HOME/bin, uncomment MODULE_OPTS add the path to the Java Agent. For Windows Platform, edit the file standalone.conf.bat editing the same MODULE_OPTS variable as Linux.

74 # Uncomment to add a Java agent. If an agent is added to the module options, then jboss-modules.jar is added as an agent
75 # on the JVM. This allows things like the log manager or security manager to be configured before the agent is invoked.
76 MODULE_OPTS="-javaagent:agent.jar" <---- add the agent path here

The agent's path will not be printed on the JBoss console log, however, it can be seen present in the sun.java.command line of the server.log file, example:

	[Standalone] = 
	awt.toolkit = sun.awt.X11.XToolkit
	file.encoding = UTF-8
	...
	java.home = /path/java-11-openjdk-11.0.9.11-2.portable.jdk.el.x86_64
	...
	sun.io.unicode.encoding = UnicodeLittle
	sun.java.command = /home/path/agent.jar -mp /jboss-eap-7.4/modules org.jboss.as.standalone -Djboss.home.dir=/jboss-eap-7.4 -Djboss.server.base.dir=/jboss-eap-7.4/standalone
	sun.java.launcher = SUN_STANDARD

Root Cause

MODULE_OPTS behavior

When you add an agent to the JAVA_OPTS the agent may initialize logging before jboss-modules does. When using MODULE_OPTS it passes the agent argument to jboss-modules instead of the java command. jboss-modules then invokes the agent after the log manager has been initialized. This was the commit that included that change into JBoss Modules via Content from github.com is not included.this commit.

In this matter MODULE_OPTS doesn't append, but instead the generated command from .e.g standalone.sh would look something like this below:

  • When added to JAVA_OPTS:

    java -javaagent:myagen.jar -Xmx512 -jar jboss-modules.jar org.jboss.as.standalone

  • With the MODULE_OPTS:

    java -Xmx512 -jar jboss-modules.jar -javaagent:myagent.jar org.jboss.as.standalone

Essentially the -javaagent is passed to jboss-modules and jboss-modules handles invoking it, which cover the majority of the cases.
In this matter, the LogManager.getLogManager() cannot be invoked before the java.util.logging.manager is set. That could be invoked in a Logger.getLogger() call. The LogManager.getLogManager() invokes a static block which sets the implementation at most once so we have to ensure the property is set. JBoss Modules handles that in it's boot process so if a logger is created before that happens, there will be problems.

Arquillian and EAP Maven Plugin

Arquilian nor EAP Maven plugin (wildfly-maven-plugin )support currently MODULE_OPTS environment variable required fro agent usage, see above.
For this matter the jiras This content is not included.Arquilian Jira 138 and This content is not included.wildfly-maven-plugin - jira 210 were created, which was relesed with the the version of Arqillian 4.0.1.Final.

FAQ on Agent injection on EAP

Q1. Does EAP support JVM on-the-fly injection?
A1. It does, but not if the agent does logging early. Example: byteman agent works fine because it doesn't log anything early, so doesn't cause the issue. User needs to be aware of that as a recommendation it is easier to pass the agent to JBoss Modules instead of on-the-fly injection.

Q2. Is there any way EAP could avoid a LocalManagementAgent connecting through jdk.internal.agent.Agent to avoid that race condition relationship to all "javaagent" modules?
A2. Not really. It's the main reason we introduced the MODULE_OPTS mechanism and invoking agents in JBoss Modules instead of having the JVM do it.

Logger

If a Java agent needs to create a java.util.logging.Logger in its premain() method, then it will need to be able to access the org.jboss.logmanager classes before JBoss Modules has been initialized and had a chance to configure the classloader for JBoss LogManager. Therefore, JBoss LogManager classes will need to be added to the boot classloader.

Once JBoss Modules has been initialized and has begun execution of the main() method of JBoss AS, JBoss LogManager is now available via a ModuleClassLoader. This will conflict with the JBoss LogManager classes loaded via the boot/system classloader, causing a ClassCastException when JBoss tries to initialize its first Logger (see the code at 1). The ClassCastException is caught and discarded in org.jboss.logmanager.Logger code, being replaced with the fatal error message:

java.lang.IllegalStateException: The LogManager was not properly installed (you must set the "java.util.logging.manager" system property to "org.jboss.logmanager.LogManager"

The solution is to make JBoss Modules use the same classloader to load JBoss LogManager classes that was used by the Java agent when it initialized the java.util.LogManager.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.