[Satellite 6] Tomcat service fails with "keystore password was incorrect" error when upgrading Red Hat Satellite

Solution Verified - Updated

Environment

Red Hat Satellite 6

Issue

  • Missing cached credential files required by the foreman-installer in /var/lib/puppet/foreman_cache_data directory.
  • Tomcat service failed to start with the following error when upgrading Satellite 6
Upgrade Step: migrate_candlepin...
Migrating candlepin database
Liquibase Update Successful
 
Upgrade Step: start_tomcat...
Redirecting to /bin/systemctl start  tomcat.service
 
Some services failed to start: tomcat
 
Upgrade step start_tomcat failed. Check logs for more information.
[root@satellite foreman-installer]# /bin/systemctl status  tomcat.service -n 300 -l
â tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2017-01-12 19:12:07 EST; 39s ago
 Main PID: 8412 (java)
   CGroup: /system.slice/tomcat.service
           ââ8412 /usr/lib/jvm/jre/bin/java -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
 
Jan 12 19:12:07 satellite systemd[1]: Started Apache Tomcat Web Application Container.
Jan 12 19:12:07 satellite systemd[1]: Starting Apache Tomcat Web Application Container...
Jan 12 19:12:07 satellite server[8412]: Java virtual machine used: /usr/lib/jvm/jre/bin/java
Jan 12 19:12:07 satellite server[8412]: classpath used: /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
Jan 12 19:12:07 satellite server[8412]: main class used: org.apache.catalina.startup.Bootstrap
...
Jan 12 19:12:08 satellite server[8412]: INFO: Initializing ProtocolHandler ["http-bio-8443"]
Jan 12 19:12:08 satellite server[8412]: Jan 12, 2017 7:12:08 PM org.apache.coyote.AbstractProtocol init
Jan 12 19:12:08 satellite server[8412]: SEVERE: Failed to initialize end point associated with ProtocolHandler ["http-bio-8443"]
Jan 12 19:12:08 satellite server[8412]: java.io.IOException: keystore password was incorrect
...
Jan 12 19:12:08 satellite server[8412]: at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:253)
Jan 12 19:12:08 satellite server[8412]: at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:427)
Jan 12 19:12:08 satellite server[8412]: Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded

Resolution

  • If the issue is occurring when upgrading a FIPS enabled Red Hat Satellite 6.14 to 6.15.0, then please refer to this solution article, otherwise continue referring to the rest of the steps mentioned below.

  • Restore the credential files in /opt/puppetlabs/puppet/cache/foreman_cache_data/ directory

  • Determine all password changes in /var/log/foreman-installer/satellite.log file.

$ grep -rin -A 10  "\-\-\-" satellite.log
--- /etc/pki/katello/private/katello-default-ca.pwd	2015-05-09 14:17:57.000000000 -0400
+++ /tmp/puppet-file20170112-27989-112v6mg	2017-01-12 16:56:27.062317706 -0500
@@ -1 +1 @@
-[OLD KATELLO CA PASSOWRD]  <======== ***PASSWORD TO BE RESTORED****
\ No newline at end of file
+[NEW KATELLO CA PASSWORD]
--
--- /etc/tomcat/server.xml	2016-08-18 19:50:47.000000000 -0400
+++ /tmp/puppet-file20170112-27989-1yc5n03	2017-01-12 16:56:28.928317795 -0500
 @@ -90,7 +90,7 @@
-               keystorePass="[OLD TOMCAT PASSWORD]"  <======== ***PASSWORD TO BE RESTORED****
+              keystorePass="[NEW TOMCAT PASSWORD]"
--
--- /etc/candlepin/candlepin.conf	2016-08-18 19:50:47.000000000 -0400
+++ /tmp/puppet-file20170112-27989-1p5h6kb	2017-01-12 16:56:28.953317796 -0500
 @@ -10,8 +10,8 @@
-candlepin.amqp.keystore_password=[OLD CANDLEPIN KEYSTORE PASSWORD]
-candlepin.amqp.truststore_password=[OLD CANDLEPIN TRUSTSTORE PASSWORD]
+candlepin.amqp.keystore_password=[NEW CANDLEPIN KEYSTORE PASSWORD]
+candlepin.amqp.truststore_password=[NEW CANDLEPIN TRUSTSTORE PASSWORD]
--
--- /etc/pulp/server.conf	2016-08-18 19:50:47.000000000 -0400
+++ /tmp/puppet-file20170112-27989-14t9mjo	2017-01-12 16:56:29.017317799 -0500
-default_password: [OLD PULP PASSWORD]  <======== ***PASSWORD TO BE RESTORED****
+default_password: [NEW PULP PASSWORD]
  • The above logs show that 3 credentials (katello, tomcat/candlepin and pulp) have been changed due to missing cache files. It is because Satellite-installer will regenerate new credentials for a service if no previously cached credential file was found in the /opt/puppetlabs/puppet/cache/foreman_cache_data/ directory.
  • To prevent this, restore the cache files in the /opt/puppetlabs/puppet/cache/foreman_cache_data/ directory
NOTE: The passwords to be restored are highlighted above.
$ cd /opt/puppetlabs/puppet/cache/foreman_cache_data/
$ echo -n "--- [OLD KATELLO CA PASSOWRD]" > ca_key_password
$ echo -n "--- [OLD TOMCAT PASSWORD]" > keystore_password-file
$ echo -n '--- "[OLD PULP PASSWORD]"' > pulp_password
  • Rerun satellite-installer --scenario satellite.

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

  • Cache files in /opt/puppetlabs/puppet/cache/foreman_cache_data/ directory have been deleted.

Diagnostic Steps

  • Check any configuration changes in /var/log/foreman-installer/satellite.log.
  • Check the following puppet file and notice that the cache_data function will re-generate a new candlepin keystore if not exists.
$ grep -rin "foreman_cache_data"  /usr/share/katello-installer-base/modules/certs/manifests/candlepin.pp
42:  $keystore_password = cache_data('foreman_cache_data', $keystore_password_file, random_password(32))
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.