JDR fails with "WFLYELY01209: Unable to initialize CredentialStore" when using encrypted system properties in JBoss EAP 8.x
Environment
- Red Hat JBoss Enterprise Application Platform (EAP) 8.x
- System properties configured with Elytron encrypted expressions (
${ENC::...}) - Management interface bound to a non-default IP address and port (other than
localhost:9990) - JBoss Diagnostic Reporter (JDR)
Issue
-
JBoss Diagnostic Reporter (JDR) fails with "WFLYELY01209: Unable to initialize CredentialStore" when using encrypted system properties in JBoss EAP 8.x
-
When running the JBoss Diagnostic Reporter (JDR) utility on JBoss EAP 8, the process fails to generate a report if it cannot connect to a running instance and attempts to start an internal embedded server.
-
This failure specifically occurs when system properties, such as
javax.net.ssl.keyStorePassword, are configured to use Elytron encrypted expressions (${ENC::...}):<system-properties> ...(snip)... <property name="javax.net.ssl.keyStorePassword" value="${ENC::...}"/> </system-properties> -
The following error is observed in the output:
ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0013: Operation ("add") failed - address: ([("system-property" => "javax.net.ssl.keyStorePassword")]) - failure description: "WFLYELY01209: Unable to initialize CredentialStore examplePropertiesCredentialStore -- WFLYELY00924: Unable to create immediately available credential store." FATAL [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0056: Server boot has failed in an unrecoverable manner; exiting. See previous messages for details. - Server configuration file in use: standalone.xml
Resolution
Red Hat is aware of this issue where the embedded server fails to resolve encrypted expressions during JDR execution. This issue has been reported and being tracked as the following JIRA:
- 8.0.z: This content is not included.JBEAP-32307 - jdr.sh fails to start embedded server when Elytron encrypted expressions is used in system-properties
- 8.1.z: This content is not included.JBEAP-32308 - jdr.sh fails to start embedded server when Elytron encrypted expressions is used in system-properties, which will be fixed in 8.1 Update 6.1 (8.1.6.1) tentatively
Note: EAP 8.0.x will not be updated beyond 8.0.12 (8.0 Update 12), which was released on March 5. This means that the issue will not be fixed for 8.0.x.
Workaround:
If the JBoss EAP instance is currently running, avoid the internal embedded server startup by explicitly pointing the JDR utility to the active management interface:
-
Identify the management IP address and port from the
standalone.xmlorserver.log(e.g.,10.0.0.1:9990). -
Execute the
jdr.shscript using the-H(host) and-p(port) flags:$ ./bin/jdr.sh -H <MANAGEMENT_IP> -p <MANAGEMENT_PORT>for example:
./bin/jdr.sh -H 10.0.0.1 -p 9990
Root Cause
-
The
jdr.shutility attempts to connect tolocalhost:9990by default. If the connection fails, it initiates an internal embedded server to collect diagnostic data; however, this embedded process is unable to initialize theCredentialStorerequired to resolve encrypted values in thesystem-propertiessection ofstandalone.xml. -
This issue has been reported and being tracked as the following JIRA:
- 8.0.z: This content is not included.JBEAP-32307 - jdr.sh fails to start embedded server when Elytron encrypted expressions is used in system-properties
- 8.1.z: This content is not included.JBEAP-32308 - jdr.sh fails to start embedded server when Elytron encrypted expressions is used in system-properties
Diagnostic Steps
-
Check the standalone.xml file for system properties using encrypted expressions. Look for values starting with
${ENC:::<system-properties> ...(snip)... <property name="javax.net.ssl.keyStorePassword" value="${ENC::...}"/> </system-properties> -
Verify the management interface binding in
standalone.xmlto see if it differs fromlocalhost. For example:<interface name="management"> <inet-address value="${jboss.bind.address.management:10.0.0.1}"/> </interface><interface name="management"> <inet-address value="${jboss.bind.address.management:exampleNode1}"/> </interface> -
Check
server.logto confirm the actual listening address:INFO [org.jboss.as] WFLYSRV0060: Http management interface listening on http://10.0.0.1:9990/management -
Review the
jdr.shconsole output forWFLYELY01209andWFLYSRV0056errors during the "Starting embedded server" phase.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.