Red Hat Satellite 6 installation fails with error 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]'

Solution Verified - Updated

Environment

  • Red Hat Satellite 6
  • Red Hat Enterprise Linux 8

Issue

  • Red Hat Satellite 6 installation on a fresh RHEL 8 server fails with the following error.

    2022-08-12 12:45:41 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-08-12 12:45:41 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-08-12 12:45:48 [NOTICE] [configure] 1500 configuration steps out of 1630 steps complete.
2022-08-12 12:45:53 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-08-12 12:45:53 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

Resolution

Increase the max_connection and shared_buffers parameter in /var/lib/pgsql/data/postgresql.conf either by:

  • adding to installer config file /etc/foreman-installer/custom-hiera.yaml below config and running installer afterwards:

    postgresql::server::config_entries:
   max_connections: 1000
   shared_buffers: 4GB

  • or applying at least medium (or [[extra-]extra-]large one(s)) tuning directly:

    satellite-installer --tuning medium

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

  • Issue with the connections being opened for Postgres process.

    # pstree | grep postmaster | wc -l
      504

  • It exhausted the default max_connection limit of 500

    $ grep max_connection var/log/foreman-installer/satellite.log 
 	2022-02-15 04:51:23 [DEBUG ] [configure] "max_connections" => 500,

Diagnostic Steps

  • DB migrate trace output shows the Postgres connection errors.

    # foreman-rake db:migrate --trace --verbose
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
Apipie cache enabled but not present yet. Run apipie:cache rake task to
speed up API calls.
rake aborted!
PG::ConnectionBad: FATAL:  remaining connection slots are reserved for
non-replication superuser connections
/usr/share/gems/gems/pg-1.2.3/lib/pg.rb:58:in `initialize'
/usr/share/gems/gems/pg-1.2.3/lib/pg.rb:58:in `new'
/usr/share/gems/gems/pg-1.2.3/lib/pg.rb:58:in `connect'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapt
ers/postgresql_adapter.rb:46:in `postgresql_connection'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapt
ers/abstract/connection_pool.rb:887:in `new_connection'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapt
ers/abstract/connection_pool.rb:931:in `checkout_new_connection'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapt
ers/abstract/connection_pool.rb:910:in `try_to_checkout_new_connection'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapt
ers/abstract/connection_pool.rb:871:in `acquire_connection'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapt
ers/abstract/connection_pool.rb:593:in `checkout'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapt
ers/abstract/connection_pool.rb:437:in `connection'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_adapt
ers/abstract/connection_pool.rb:1125:in `retrieve_connection'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_handl
ing.rb:221:in `retrieve_connection'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/connection_handl
ing.rb:189:in `connection'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/model_schema.rb:
517:in `load_schema!'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/attributes.rb:24
7:in `load_schema!'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/attribute_decora
tors.rb:50:in `load_schema!'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/model_schema.rb:
507:in `block in load_schema'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/model_schema.rb:
504:in `synchronize'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/model_schema.rb:
504:in `load_schema'
/usr/share/gems/gems/activerecord-6.0.4.7/lib/active_record/model_schema.rb:
381:in `columns_hash'
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.