[Satellite 6] Capsule synchronization fails with deadlock detected even on 6.12.1+

Environment

Red Hat Satellite/Capsule 6.12.1+

Issue

Even on Satellite/Capsule upgraded to 6.12.1 with the errata RHSA-2023:0261 applied, Capsule sync gets deadlocked, with deadlock like:

DETAIL:  Process 7102 waits for ShareLock on transaction 303148; blocked by process 7128.
Process 7128 waits for ShareLock on transaction 303147; blocked by process 7102.
HINT:  See server log for query details.
CONTEXT:  while locking tuple (1430,5) in relation "rpm_package"
)

Resolution

For a fix, wait until the This content is not included.underlying bugzilla is fixed.

As a workaround, follow the remedy steps from this solution - same logic and same remedy steps apply.

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

Diagnostic Steps

Backtrace of one of the deadlocked process:

File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 568, in synchronize
    repo_version = dv.create() or repo.latest_version()
  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/declarative_version.py", line 161, in create
    loop.run_until_complete(pipeline)
  File "/usr/lib64/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
    await asyncio.gather(*futures)
  File "/usr/lib/python3.9/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
    await self.run()
  File "/usr/lib/python3.9/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 1623, in run
    await sync_to_async(result.save)()
  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 414, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/lib64/python3.9/asyncio/tasks.py", line 442, in wait_for
    return await fut
  File "/usr/lib64/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3.9/site-packages/asgiref/sync.py", line 455, in thread_handler
    return func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/pulpcore/app/models/base.py", line 173, in save
    return super().save(*args, **kwargs)
  File "/usr/lib64/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3.9/site-packages/django_lifecycle/mixins.py", line 134, in save
    save(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/usr/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/usr/lib/python3.9/site-packages/django/db/models/base.py", line 858, in _save_table
    updated = self._do_update(base_qs, using, pk_val, values, update_fields,
  File "/usr/lib/python3.9/site-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
  File "/usr/lib/python3.9/site-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/usr/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/usr/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.