Getting Invalid secret key format when starts up the server or accessing to a vault in JBoss EAP

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6
    • 7
  • Red Hat JBoss Data Grid
    • 7.1.2
  • JDK that has included non-public JDK-8189997 including:
    • JDK 1.8.0_171+ (OpenJDK or Oracle JDK)
    • JDK 1.7.0_181+ (OpenJDK or Oracle JDK)
  • Using Elytron Credential Stores or the JBoss EAP Vault
  • Often after version migration or java update

Issue

  • Exception thrown or Elytron unable to initialize:

      Caused by: java.io.IOException: Invalid secret key format
  	at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:856)
  	at java.security.KeyStore.load(KeyStore.java:1445)
  	at org.picketbox.util.KeyStoreUtil.getKeyStore(KeyStoreUtil.java:201)
  	at org.picketbox.util.KeyStoreUtil.getKeyStore(KeyStoreUtil.java:151)
  	at org.picketbox.plugins.vault.PicketBoxSecurityVault.getKeyStore(PicketBoxSecurityVault.java:688)
  	... 11 more

    or

        Caused by: java.io.IOException: Invalid secret key format
    at com.sun.crypto.provider.JceKeyStore.engineLoad(JceKeyStore.java:856)
    at java.security.KeyStore.load(KeyStore.java:1445) [rt.jar:1.8.0_191-1-redhat]
    at org.wildfly.security.credential.store.impl.KeyStoreCredentialStore.load(KeyStoreCredentialStore.java:859) [wildfly-elytron-1.6.1.Final-redhat-00001.jar:1.6.1.Final-redhat-00001]
    ... 11 more

  • An ObjectInputFilter REJECTED error has been thrown when trying to create a Vault:

      WFLYSEC0056: Initializing Vault
  May 01, 2018 11:13:49 AM java.io.ObjectInputStream filterCheck
  INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 1, depth: 1, bytes: 70, ex: n/a
  WFLYSEC0059: Exception encountered:WFLYSEC0045: Exception encountered:

  • NullPointerException in log:

      Caused by: org.jboss.security.vault.SecurityVaultException: java.lang.NullPointerException
         at org.picketbox.plugins.vault.PicketBoxSecurityVault.init(PicketBoxSecurityVault.java:192)
         at org.jboss.as.security.vault.RuntimeVaultReader.createVault(RuntimeVaultReader.java:82) [jboss-as-security-7.3.0.Final-redhat-14.jar:7.3.0.Final-redhat-14]
         ... 11 more
   Caused by: java.lang.NullPointerException
         at org.picketbox.plugins.vault.PicketBoxSecurityVault.checkAndConvertKeyStoreToJCEKS(PicketBoxSecurityVault.java:527)
         at org.picketbox.plugins.vault.PicketBoxSecurityVault.init(PicketBoxSecurityVault.java:189)
         ... 12 more

  • Server doesn't start because it can't find the vault file.

Resolution

This is a known issue, see This content is not included.Bugzilla ticket 1567126, the fix is included in JBoss EAP 7.1 CP3 and EAP 6.4 CP21.

There is a Workaround for versions before JBoss EAP 7.1 CP3 and JBoss EAP 6.4 CP21. Add the com.sun.crypto.provider to the JBOSS_MODULES_SYSTEM_PKGS (which in turn adds the value to the system property jboss.modules.system.pkgs) in the standalone.conf or standalone.bat, since -Djboss.modules.system.pkgs is a JVM parameter not a OS parameter.

  1. Locate the relevant startup file.

    • Red Hat Enterprise Linux (RHEL) 6 or 7

      • Domain Mode: JBOSS_HOME/bin/domain.conf
      • Standalone Mode: JBOSS_HOME/bin/standalone.conf

    • Microsoft Windows Server

      • Domain Mode: JBOSS_HOME/bin/domain.conf.bat
      • Standalone Mode: JBOSS_HOME/bin/standalone.conf.bat

  2. Add "com.sun.crypto.provider" to "jboss.modules.system.pkgs"

    • For the ".conf" files on RHEL, you'll find a variable named JBOSS_MODULES_SYSTEM_PKGS

        JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,com.sun.crypto.provider"

    • In the .bat files, it will just be added to the JAVA_OPTS

        set "JAVA_OPTS=%JAVA_OPTS% -Djboss.modules.system.pkgs=org.jboss.byteman,com.sun.crypto.provider"

  3. Verify after starting JBoss EAP. For example, you can check the startup logs for the command line options, or on RHEL run:

    ps -ef | grep jboss.modules.system.pkgs

    This will give process output and you should be able to see the command line args.

  4. This issue also happens in the vault.sh or vault.bat script but this command does not read the conf file (standalone.conf or domain.conf) so, until the patch is released, use JAVA_OPTS to pass the jboss.modules.system.pkgs system property when executing the vault.sh, for example:

    JAVA_OPTS="-Djboss.modules.system.pkgs=com.sun.crypto.provider" ${JBOSS_HOME}/bin/vault.sh --keystore ${JBOSS_HOME}/vault/vault.keystore --keystore-password XXXXX --alias vault --check-sec-attr --attribute password --vault-block vb --enc-dir ${JBOSS_HOME}/vault

    make sure to set JBOSS_HOME to the root of the JBoss EAP installation.

    JBoss EAP 7.1 already has the dependency in the Vault module but it can be affected in other modules that use a JCEKS keystore. JBoss EAP 6.4 and 7.0 are affected in all the presents patches.

Root Cause

The bug fix is jboss-modules adding com.sun.crypto.provider package internally so the following workaround is required for versions after these.

The root cause for this is RHSA-2018:1191 managed in bugzilla This content is not included.1567126 which protected how the JCEKS keystore loads its keys in openjdk. As a result of that, a new dependency is needed to read the class com.sun.crypto.provider.SealedObjectForKeyProtector (finally it was decided to add that dependency globally to jboss-modules).

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.