Upgrade to Red Hat Satellite 6.10 fails with error '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.10

Issue

  • While upgrading to Red Hat Satellite 6.10, It fails with the following error in satellite.log:

     2022-03-09 09:21:42 [DEBUG ] [configure] Exec[foreman-rake-db:migrate](provider=posix): Executing '/usr/sbin/foreman-rake db:migrate'
 2022-03-09 09:21:42 [DEBUG ] [configure] Executing with uid=foreman: '/usr/sbin/foreman-rake db:migrate'
 2022-03-09 09:22:10 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: 'ErbParser' is ignored.
 2022-03-09 09:22:10 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: 'RubyParser' is ignored.
 2022-03-09 09:22:10 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: API controllers newer than Api
  pie cache! Run apipie:cache rake task to regenerate cache.
2022-03-09 09:22:10 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
2022-03-09 09:22:10 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: StandardError: An error has oc
 curred, this and all later migrations canceled:
 2022-03-09 09:22:10 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns:
 2022-03-09 09:22:10 [INFO  ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: PG::ForeignKeyViolation: ERROR:  update or delete 
 on table "katello_repositories" violates foreign key constraint "katello_repository_errata_repo_id_fk" on table "katello_repository_errata"

Resolution

This problem was reported to the Red Hat Engineering team via Bugzilla This content is not included.2038192 and This content is not included.2070996 and further has been fixed on Red Hat Satellite 6.10.5 via the errata RHSA-2022:1708.

 

Workaround: ( When the target version to upgrade the satellite is 6.10.4 or below )

  • If the traceback of the error is the same as mentioned in the Diagnostic Steps section, then to solve the problem:

    • Find out if there are any ERRATA's present that is related to OStree and\or Puppet type repositories.

      # echo "pp Katello::RepositoryErratum.where(:repository_id => Katello::Repository.where(:root_id => Katello::RootRepository.where(:content_type => [:ostree, :puppet])))" | foreman-rake console

    • If the above command returns some output, Then clear up that data and verify that no such content exists now.

             # echo "Katello::RepositoryErratum.where(:repository_id => Katello::Repository.where(:root_id => Katello::RootRepository.where(:content_type => [:ostree, :puppet]))).delete_all" | foreman-rake console

       # echo "pp Katello::RepositoryErratum.where(:repository_id => Katello::Repository.where(:root_id => Katello::RootRepository.where(:content_type => [:ostree, :puppet])))" | foreman-rake console

    • Perform the db:migrate command manually and ensure that it is getting completed successfully.

      # foreman-rake db:migrate --trace --verbose

    • Once successful, Re-run the upgrade of the Satellite server.

 

Please reach out to This content is not included.Red Hat Technical Support in case of any further concerns or assistance required.

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Diagnostic Steps

  • The manual execution of db:migrate command shows the following error.

    # foreman-rake db:migrate --trace --verbose
'ErbParser' is ignored.
'RubyParser' is ignored.
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
** Execute db:load_config
** Invoke plugin:refresh_migrations (first_time)
** Invoke environment
** Execute plugin:refresh_migrations
** Execute db:migrate
== 20210119162528 DeletePuppetAndOstreeRepos: migrating =======================
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::ForeignKeyViolation: ERROR:  update or delete on table "katello_repositories" violates foreign key constraint "katello_repository_errata_repo_id_fk" on table "katello_repository_errata"
DETAIL:  Key (id)=(205) is still referenced from table "katello_repository_errata".
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `async_exec_params'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:675:in `block (2 levels) in exec_no_cache'
...
...
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:587:in `delete_all'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation.rb:616:in `delete_by'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/querying.rb:21:in `delete_by'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/persistence.rb:352:in `delete'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/db/migrate/20210119162528_delete_puppet_and_ostree_repos.rb:44:in `block in up'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.1.1.39/db/migrate/20210119162528_delete_puppet_and_ostree_repos.rb:43:in `up'
...
...

  • Following command reveals that there are some ERRATA present indeed referencing to OStree and\or Puppet type repositories.

       #  echo "Katello::RepositoryErratum.where(:repository_id => Katello::Repository.where(:root_id => Katello::RootRepository.where(:content_type => [:ostree, :puppet])))" | foreman-rake console

   'ErbParser' is ignored.
   'RubyParser' is ignored.
   API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
   Loading production environment (Rails 6.0.3.7)
   Switch to inspect mode.
   Katello::RepositoryErratum.where(:repository_id => Katello::Repository.where(:root_id => Katello::RootRepository.where(:content_type => [:ostree, :puppet])))
   #<ActiveRecord::Relation [#<Katello::RepositoryErratum id: 4334583, erratum_id: 14645, repository_id: 205, created_at: "2022-01-02 22:50:12", updated_at: "2022-01-02 22:50:12", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/ddbb01dd-dbac-...">, #<Katello::RepositoryErratum id: 4334572, erratum_id: 14755, repository_id: 205, created_at: "2022-01-02 22:50:12", updated_at: "2022-01-02 22:50:12", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/93e99769-9f00-...">, #<Katello::RepositoryErratum id: 4334561, erratum_id: 14513, repository_id: 205, created_at: "2022-01-02 22:50:12", updated_at: "2022-01-02 22:50:12", erratum_pulp3_href: "/pulp/api/v3/content/rpm/advisories/047078a2-cc48-...">,  ...]>
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.