Red Hat Satellite or Capsule upgrade 6.12 to 6.13 fails with error `pulpcore-manager migrate --noinput' returned 1 instead of one of [0]`

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.12
  • Red Hat Satellite Capsule 6.12
  • Red Hat Enterprise Linux 8

Issue

  • Executing, satellite-maintain upgrade run --target-version 6.13 returns the following error:

    [NOTICE] [configure] 2750 configuration steps out of 3100 steps complete.
[ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
[ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
[NOTICE] [configure] System configuration has finished.
...
  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/satellite.log
Package versions are being locked.
                                      [FAIL]
Failed executing satellite-installer, exit status 6
--------------------------------------------------------------------------------
Scenario [Migration scripts to Satellite 6.13] failed.

Resolution

  • Issue is currently being tracked via This content is not included.Bugzilla 2203353

  • Ensure you have a working backup or snapshot taken of the Satellite before proceeding

  • Execute the below on the Satellite server:

    1. Ensure satellite services and any potential pulpcore-worker not managed by Satellite are stopped:

      # satellite-maintain service stop
# systemctl stop pulpcore-worker@multi-user.service

    2. Perform the upgrade:

      # satellite-maintain upgrade run --target-version 6.13

If the same deadlock is repeatedly hit, let try removing some old pulp tasks to decrease probability of hitting the deadlock on updating tasks:

weekago="2024-08-14"  # set to a date approx week ago

echo "Task.objects.filter(finished_at__lt='${weekago}').delete()" | sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell

Then re-run the failing upgrade step again.

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

  • Deadlock was detected on the database which was causing the upgrade to result in an exit status code 6 or failure.
  • Alive pulpcore-worker process using the database during migration

Diagnostic Steps

  • Verify the upgrade check completes with all [OK]s, by executing:

    satellite-maintain upgrade check --target-version 6.13

  • Check /var/log/foreman-installer/satellite.log for errors like:

    [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
[ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
[NOTICE] [configure] System configuration has finished.

  • Check /var/lib/pgsql/data/log/postgresql-DAY.log -- Search the log for the day the upgrade was executed for errors like below:

    2023-05-08 15:48:21 EDT LOG:  duration: 21045.299 ms  statement: ALTER TABLE "core_task" ALTER COLUMN "reserved_resources
_record" TYPE text[] USING "reserved_resources_record"::text[]
2023-05-08 15:48:22 EDT ERROR:  deadlock detected
2023-05-08 15:48:22 EDT DETAIL:  Process 17216 waits for AccessExclusiveLock on relation 1373490 of database 647534; blocked by process 11610.
        Process 11610 waits for AccessShareLock on relation 647792 of database 647534; blocked by process 17216.
        Process 17216: DROP INDEX IF EXISTS "core_taskschedule_name_f9ade8e8_like"
        Process 11610: SELECT "core_taskschedule"."pulp_id", "core_taskschedule"."pulp_created", "core_taskschedule"."pulp_last_updated", "core_taskschedule"."name", "core_taskschedule"."next_dispatch", "core_taskschedule"."dispatch_interval", "core_taskschedule"."task_name", "core_taskschedule"."last_task_id" FROM "core_taskschedule" LEFT OUTER JOIN "core_task" ON ("core_taskschedule"."last_task_id" = "core_task"."pulp_id") WHERE ("core_taskschedule"."next_dispatch" <= '2023-05-08T19:48:07.457888+00:00'::timestamptz AND ("core_taskschedule"."last_task_id" IS NULL OR "core_task"."state" IN ('skipped', 'completed', 'failed', 'canceled')))

  • Execute the following command, PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager migrate --noinput to check for django.db.utils.OperationalError: deadlock detected and/or psycopg2.errors.DeadlockDetected: deadlock detected:

       # PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager migrate --noinput

Operations to perform:
  Apply all migrations: ansible, auth, certguard, container, contenttypes, core, file, rpm, sessions
Running migrations:
  Applying core.0090_char_to_text_field...Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.DeadlockDetected: deadlock detected
DETAIL:  Process 31321 waits for AccessExclusiveLock on relation 1373490 of database 647534; blocked by process 11607.
Process 11607 waits for AccessShareLock on relation 647792 of database 647534; blocked by process 31321.
HINT:  See server log for query details.

  • The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
  File "/usr/bin/pulpcore-manager", line 33, in <module>
    sys.exit(load_entry_point('pulpcore==3.21.6', 'console_scripts', 'pulpcore-manager')())
  File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/fields.py", line 244, in database_forwards
    schema_editor.alter_field(from_model, from_field, to_field)
  File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 608, in alter_field
    self._alter_field(model, old_field, new_field, old_type, new_type,
  File "/usr/lib/python3.9/site-packages/django/db/backends/postgresql/schema.py", line 194, in _alter_field
    self.execute(self._delete_index_sql(model, index_name))
  File "/usr/lib/python3.9/site-packages/django/db/backends/base/schema.py", line 145, in execute
    cursor.execute(sql, params)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.OperationalError: deadlock detected
DETAIL:  Process 31321 waits for AccessExclusiveLock on relation 1373490 of database 647534; blocked by process 11607.
Process 11607 waits for AccessShareLock on relation 647792 of database 647534; blocked by process 31321.
HINT:  See server log for query details.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.