Upgrading a FIPS enabled Red Hat Satellite 6.14 to 6.15 fails with error 'java.io.IOException: keystore password was incorrect'

Solution Verified - Updated 30 Jun 2024

Environment

Red Hat Satellite 6.15.0 ( being upgraded from Red Hat Satellite 6.14.z )
FIPS enabled

Issue

When upgrading a Red Hat Satellite 6.14 to 6.15.0 that has FIPS mode enabled, The installer execution fails with the following set of errors:

   2024-04-25 15:29:16 [NOTICE] [configure] 1500 configuration steps out of 1622 steps complete.
   2024-04-25 15:29:35 [NOTICE] [configure] System configuration has finished.

   Error 1: Puppet Truststore_certificate resource '/etc/candlepin/certs/truststore:artemis-client' failed. Logs:
   ...
   ...
       Starting to evaluate the resource (661 of 1613)
       Evaluated in 0.52 seconds
     /Stage[main]/Certs::Candlepin/Truststore_certificate[/etc/candlepin/certs/truststore:artemis-client]/ensure
    change from 'absent' to 'present' failed: Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12-keystore /etc/candlepin/certs/truststore -alias artemis-client -file /etc/foreman/client_cert.pem -storepass:file /etc/pki/katello/truststore_password-file -J-Dcom.redhat.fips=false' returned 1: keytool error: java.io.IOException: keystore password was incorrect

   ..
   ..
    Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	   ... 5 more
   Error 2: Puppet Truststore_certificate resource '/etc/candlepin/certs/truststore:candlepin-ca' failed. Logs:
  /Stage[main]/Certs::Candlepin/Truststore_certificate[/etc/candlepin/certs/truststore:candlepin-ca]
   ..
   ..
     /Stage[main]/Certs::Candlepin/Truststore_certificate[/etc/candlepin/certs/truststore:candlepin-ca]/ensure
    change from 'absent' to 'present' failed: Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias candlepin-ca -file /etc/candlepin/certs/candlepin-ca.crt -storepass:file /etc/pki/katello/truststore_password-file -J-Dcom.redhat.fips=false' returned 1: keytool error: java.io.IOException: keystore password was incorrect
   ..
   ..
   Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
	   ... 5 more

   2 errors were detected.
   Please address the errors and re-run the installer to ensure the system is properly configured.
   Failing to do so is likely to result in broken functionality.

   The full log is at /var/log/foreman-installer/satellite.log
   Package versions are being locked.
                                         [FAIL]
   Failed executing satellite-installer, exit status 6.

Resolution

This issue has been reported to the Red Hat Engineering team via the This content is not included.Bugzilla 2277005 and is being actively investigated.

Until an official fix is released, apply the following workaround to complete the upgrade process.
- Remove the old keystore and truststore files of candlepin from the affected satellite server.
```
 # mv /etc/candlepin/certs/truststore /etc/candlepin/certs/keystore /tmp/ 
```
- Re-run the upgrade procedure i.e.
```
     For connected Satellite server:

     # satellite-maintain upgrade run --target-version 6.15

     For disconnected Satellite server:

     # satellite-maintain upgrade run --target-version 6.15 --whitelist="repositories-validate,repositories-setup"
```
- Verify the application's health using the hammer ping command.

Reach out to the This content is not included.Red Hat Technical Support in case any further assistance is required or concerns to be reported.

Diagnostic Steps

The following errors are visible in the /var/log/foreman-installer/satellite.log file.

   2024-04-25 15:28:22 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Truststore_certificate[/etc/candlepin/certs/truststore:artemis-client]: Starting to evaluate the resource (661 of 1613)
   2024-04-25 15:28:22 [DEBUG ] [configure] Executing: '/bin/keytool -list -keystore /etc/candlepin/certs/truststore -storepass:file /etc/pki/katello/truststore_password-file -alias artemis-client -J-Dcom.redhat.fips=false'
   2024-04-25 15:28:22 [DEBUG ] [configure] Failed to read truststore contents: Execution of '/bin/keytool -list -keystore /etc/candlepin/certs/truststore -storepass:file /etc/pki/katello/truststore_password-file -alias artemis-client
 -J-Dcom.redhat.fips=false' returned 1: keytool error: java.io.IOException: keystore password was incorrect
   2024-04-25 15:28:22 [DEBUG ] [configure] Executing: '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias artemis-client -file /etc/foreman/client_cert.pem -storepass:file /etc/pki/ka
tello/truststore_password-file -J-Dcom.redhat.fips=false'
   2024-04-25 15:28:22 [ERROR ] [configure] Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias artemis-client -file /etc/foreman/client_cert.pem -storepass:file /etc/pki/katello/truststore_password-file -J-Dcom.redhat.fips=false' returned 1: keytool error: java.io.IOException: keystore password was incorrect
   2024-04-25 15:28:22 [ERROR ] [configure] java.io.IOException: keystore password was incorrect
   2024-04-25 15:28:22 [ERROR ] [configure] at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2089)
   2024-04-25 15:28:22 [ERROR ] [configure] at java.security.KeyStore.load(KeyStore.java:1445)
   2024-04-25 15:28:22 [ERROR ] [configure] at sun.security.tools.keytool.Main.doCommands(Main.java:839)
   2024-04-25 15:28:22 [ERROR ] [configure] at sun.security.tools.keytool.Main.run(Main.java:380)
   2024-04-25 15:28:22 [ERROR ] [configure] at sun.security.tools.keytool.Main.main(Main.java:373)
   2024-04-25 15:28:22 [ERROR ] [configure] Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
   2024-04-25 15:28:22 [ERROR ] [configure] ... 5 more
   2024-04-25 15:28:22 [ERROR ] [configure] /Stage[main]/Certs::Candlepin/Truststore_certificate[/etc/candlepin/certs/truststore:artemis-client]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias artemis-client -file /etc/foreman/client_cert.pem -storepass:file /etc/pki/katello/truststore_password-file -J-Dcom.redhat.fips=false' returned 1: keytool error: java.io.IOException: keystore password was incorrect
   2024-04-25 15:28:22 [ERROR ] [configure] java.io.IOException: keystore password was incorrect
   2024-04-25 15:28:22 [ERROR ] [configure] at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2089)
   2024-04-25 15:28:22 [ERROR ] [configure] at java.security.KeyStore.load(KeyStore.java:1445)
   2024-04-25 15:28:22 [ERROR ] [configure] at sun.security.tools.keytool.Main.doCommands(Main.java:839)
   2024-04-25 15:28:22 [ERROR ] [configure] at sun.security.tools.keytool.Main.run(Main.java:380)
   2024-04-25 15:28:22 [ERROR ] [configure] at sun.security.tools.keytool.Main.main(Main.java:373)
   2024-04-25 15:28:22 [ERROR ] [configure] Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
   2024-04-25 15:28:22 [ERROR ] [configure] ... 5 more
   2024-04-25 15:28:22 [DEBUG ] [configure] /Stage[main]/Certs::Candlepin/Truststore_certificate[/etc/candlepin/certs/truststore:artemis-client]: Evaluated in 0.52 seconds

Confirm that FIPS is enabled for the affected Red Hat Satellite server.
```
# update-crypto-policies --show
FIPS
```

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.