Upgrading to Satellite 6.12 fails on 0044_noartifact_modules migration step during "pulpcore-manager migrate --noinput"

Solution Verified - Updated 17 May 2024

Environment

Red Hat Satellite 6.11 -> 6.12 upgrade

Issue

Upgrade of Satellite 6.11 to 6.12 fails during pulpcore-manager migrate --noinput. Logs show that 0044_noartifact_modules migration step failed.

Resolution

Replace /usr/lib/python3.9/site-packages/pulp_rpm/app/migrations/0044_noartifact_modules.py on Satellite by the attached file:

cp 0044_noartifact_modules_quickfix.py_.txt /usr/lib/python3.9/site-packages/pulp_rpm/app/migrations/0044_noartifact_modules.py

chmod a+r /usr/lib/python3.9/site-packages/pulp_rpm/app/migrations/0044_noartifact_modules.py

Start the upgrade again.

Once it successfully completes, missing modularity data must be fixed by running Complete Sync of all affected repositories. To identify such repos, run:

for t in rpm_modulemd rpm_modulemddefaults; do su - postgres -c "psql pulpcore -c \"COPY (SELECT r.name FROM core_repository AS r INNER JOIN core_repositorycontent AS crc ON r.pulp_id = crc.repository_id INNER JOIN ${t} AS rmd ON crc.content_id = rmd.content_ptr_id WHERE rmd.snippet = '' OR rmd.snippet IS NULL) TO STDOUT;\""; done | sort -u

It will print line(s) like:

Red_Hat_Enterprise_Linux_8_for_x86_64_-_AppStream_RPMs_8-1857889

where the repository name will be evident.

For each such repository, run Complete Sync: i.e. go to WebUI, open the repository, click to Select Action button, select Advance Sync, chose Complete Sync and click to Sync.

Once all such synchronizations are completed, re-run the above command - it should not return any repository.

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

The failing migration script assumes data integrity in pulpcore database. When the DB lacks some modularity information, the script fails.

Updated script version does not migrate data when they are missing, and leaves fixing the missing data post upgrade.

Diagnostic Steps

/var/log/foreman-installer/satellite.log having:

2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Running migrations:
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   Applying rpm.0044_noartifact_modules...Traceback (most recent call last):
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/bin/pulpcore-manager", line 33, in <module>
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     sys.exit(load_entry_point('pulpcore==3.18.25', 'console_scripts', 'pulpcore-manager')())
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     execute_from_command_line(sys.argv)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     utility.execute()
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self.fetch_command(subcommand).run_from_argv(self.argv)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self.execute(*args, **cmd_options)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     output = self.handle(*args, **options)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     res = handle_func(*args, **kwargs)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     post_migrate_state = executor.migrate(
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     state = migration.apply(state, schema_editor)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self.code(from_state.apps, schema_editor)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/pulp_rpm/app/migrations/0044_noartifact_modules.py", line 17, in convert_artifact_to_snippets
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     artifact = module._artifacts.get()
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     return getattr(self.get_queryset(), name)(*args, **kwargs)
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 435, in get
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     raise self.model.DoesNotExist(
2023-12-22 12:23:19 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: __fake__.Artifact.DoesNotExist: Artifact matching query does not exist.
2023-12-22 12:23:19 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2023-12-22 12:23:19 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

SBR

SysMgmt

Product(s)

Red Hat Satellite

Components

pulp

Category

Upgrade

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.