Candlepin fails to start with psql authentication error despite passwords are OK

Environment

• Red Hat Satellite 6

Issue

• After manually editing or restoring /etc/candlepin/candlepin.conf file, double-checking it has the original content, an attempt to start candlepin / tomcat service fails with error on authenticating to psql. While the database password is certainly the correct one.

Resolution

• Ensure proper ownership of /etc/candlepin/candlepin.conf file - it should be tomcat:tomcat.

For more KB articles/solutions related to Red Hat Satellite 6.x Candlepin Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Candlepin Issues

Root Cause

When the config file is not visible by tomcat user that attempts to read it, some dummy password is used to authenticate to postgres DB instead the right one. Which leads to a very confusing / misleading error.

Diagnostic Steps

• /var/log/candlepin/candlepin.log contains:

  2023-03-23 19:23:22,195 [thread=main] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Registering job: UnmappedGuestEntitlementCleanerJob: org.candlepin.async.tasks.UnmappedGuestEntitlementCleanerJob
  2023-03-23 19:23:55,302 [thread=C3P0PooledConnectionPoolManager[identityToken->2tmgdjav4cxp0e1t7klna|5cee664b]-HelperThread-#0] [=, org=, csid=] WARN  com.mchange.v2.resourcepool.BasicResourcePool - com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@62f8e3e1 -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
  org.postgresql.util.PSQLException: FATAL: password authentication failed for user "candlepin"
  	at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:646)
  	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:180)
  	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:235)
  	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
  	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:223)
  	at org.postgresql.Driver.makeConnection(Driver.java:400)
  	at org.postgresql.Driver.connect(Driver.java:259)
  	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:175)
  	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:220)
  	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:206)
  	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:203)
  	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1138)
  	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1125)
  	at com.mchange.v2.resourcepool.BasicResourcePool.access$700(BasicResourcePool.java:44)
  	at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1870)
  	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:696)
  

• While /etc/candlepin/candlepin.conf has proper passwords that match how candlepin DB was initialized via cpdb command by satellite-installer:

  # grep password /etc/candlepin/candlepin.conf 
  jpa.config.hibernate.connection.password=aYoyX3hDRZZx8uA2pWRer2Ar5E4RKevh
  org.quartz.dataSource.myDS.password=aYoyX3hDRZZx8uA2pWRer2Ar5E4RKevh
  #
  # grep cpdb /var/log/foreman-installer/*
  /var/log/foreman-installer/satellite.log:2023-03-22 13:28:41 [DEBUG ] [configure] Executing: '/usr/share/candlepin/cpdb --update --dbhost=localhost --dbport=5432 --database=candlepin --user=candlepin --password=aYoyX3hDRZZx8uA2pWRer2Ar5E4RKevh'
  #
  

• But ownership of the file is wrong - it must be tomcat:tomcat:

  # ll /etc/candlepin/candlepin.conf 
  -rw-r-----. 1 root root 1878 Mar 22 19:50 /etc/candlepin/candlepin.conf
  #
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.