Impossible to re-enable puppet plugin if it was disabled and removed using -f or --remove-all-data option

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.11 - 6.17

Issue

  • It is not possible to re-enable the puppet-plugin after it was disabled and purged forcefully.

  • Attempt to re-enabling it fails satellite-installer on the db:migrate state.

    2022-05-17 04:45:47 [NOTICE] [configure] 1250 configuration steps out of 2097 steps complete.
2022-05-17 04:46:39 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2022-05-17 04:46:39 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

    The db:migrate execution itself fails with the following error:

    == 20121018152459 CreateHostgroupClasses: migrating ===========================
-- rename_table(:hostgroups_puppetclasses, :hostgroup_classes)
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::UndefinedTable: ERROR:  relation "hostgroups_puppetclasses" does not exist

Resolution

Ensure the availability of a full working backup or VM snapshot of the affected Red Hat Satellite server before proceeding with the steps mentioned below. The steps involve restarting Red Hat Satellite's services and hence should be executed within a planned maintenance window.

 

To fix the issue on the affected Red Hat Satellite server:

  1. Ensure that all services are healthy and running fine. 

    # satellite-maintain service status -b
# hammer ping

  2. Delete the puppet_facets related tables and remove the expected migration entries from the schema_migration table using the following command:

    # cat << EOF |foreman-rake console
User.current = User.anonymous_admin
ActiveRecord::Base.connection.drop_table(:hostgroup_puppet_facets, if_exists: true, force: :cascade)
ActiveRecord::Base.connection.drop_table(:host_puppet_facets, if_exists: true, force: :cascade)
ActiveRecord::SchemaMigration.where(version: %w[20090905150132 20101121140000 20201125113903 20200803113803 20200722171017 20200803113531 20200803113903 20211111125003]).delete_all
EOF

  3. Confirm that no puppet-related facets tables exist in the foreman database. 

    # echo "\dt" | su - postgres -c "psql foreman" | grep puppet_facets

  4. Unset puppet capsule and puppet ca capsule from all the hosts and hostgroups ( if any are selected ):

    # cat << EOF | foreman-rake console
User.current = User.anonymous_admin
::Hostgroup.update_all(puppet_proxy_id: nil) if ActiveRecord::Base.connection.column_exists?(:hostgroups, :puppet_proxy_id)
::Hostgroup.update_all(puppet_ca_proxy_id: nil) if ActiveRecord::Base.connection.column_exists?(:hostgroups, :puppet_ca_proxy_id)
Host::Managed.update_all(puppet_proxy_id: nil) if ActiveRecord::Base.connection.column_exists?(:hosts, :puppet_proxy_id)
Host::Managed.update_all(puppet_ca_proxy_id: nil) if ActiveRecord::Base.connection.column_exists?(:hosts, :puppet_ca_proxy_id)
EOF

  5. If all steps have been successfully executed till now, The database should be clear now. Try re-enabling puppet-plugin back on the affected satellite:

    • For Red Hat Satellite 6.11, 6.12 and 6.13:
         # satellite-installer --enable-foreman-plugin-puppet \
     --enable-foreman-cli-puppet \
     --foreman-proxy-puppet true \
     --foreman-proxy-puppetca true \
     --foreman-proxy-content-puppet true \
     --enable-puppet \
     --puppet-server true \
     --puppet-server-foreman-ssl-ca /etc/pki/katello/puppet/puppet_client_ca.crt \
     --puppet-server-foreman-ssl-cert /etc/pki/katello/puppet/puppet_client.crt \
     --puppet-server-foreman-ssl-key /etc/pki/katello/puppet/puppet_client.key

     # satellite-maintain service restart
   
     # hammer capsule info --id 1
    • For Red Hat Satellite 6.14+:
      # satellite-installer \
  --enable-foreman-cli-puppet \
  --enable-foreman-plugin-puppet \
  --enable-puppet \
  --foreman-proxy-puppet true \
  --foreman-proxy-puppetca true \
  --puppet-server true

  # satellite-maintain service restart

  # hammer capsule info --id 1

If this steps fails with error:

        == 20201125113903 MigratePuppetclassesToFacets: migrating =====================
    -- add_reference(:host_classes, :host_puppet_facet, {:foreign_key=>true, :index=>true, :_uses_legacy_reference_index_name=>true})
    change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

then a properly executed migration was marked as not done - let add it back:

su - postgres -c "psql foreman -c \"INSERT INTO schema_migrations VALUES (20201125113903);\""

and re-run the same installer command.

 

NOTE: In case the plugin needs not to be re-enabled but has to be completely removed, then it's also important to complete steps 1 - 4 of the resolution section and then restart satellite services.

Reach out to This content is not included.Red Hat Technical Support in case of any further concerns or additional assistance would be required.

For more KB articles/solutions related to Red Hat Satellite 6.x Puppet Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Puppet Issues

Root Cause

The issue has been reported to the Red Hat Engineering team via This content is not included.Bugzilla 2087067 and is under active investigation.

When satellite-maintain plugin purge-puppet is executed with -f or --remove-all-data, even though it wipes out almost every trace of puppet-plugin, It still leaves some migration entries intact in the foreman database and also leaves two puppet-related tables untouched.

Due to the same, When someone tries to enable the puppet-plugin, the Installer tries to execute a few steps whose dependencies are not recreated yet but the presence of the existing db migration entries forces the satellite to think otherwise here. Hence additional cleanup of those existing entries is required before the puppet-plugin can be gracefully re-enabled.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.