Red Hat Satellite Capsule synchronization fails with deadlock detected errors.

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.10
  • Red Hat Satellite 6.11
  • Red Hat Satellite 6.12
  • Red Hat Satellite 6.13

Issue

  • Capsule Synchronization fails with DeadLock detected on table core_artifact or core_contentartifact.

      Task errors: deadlock detected
  DETAIL:  Process 00001 waits for ShareLock on transaction 1000001; blocked by process 00001.
  Process 00001 waits for ShareLock on transaction 1000001; blocked by process 00002.
  HINT:  See server log for query details.
  CONTEXT:  while rechecking updated tuple (0001,1) in relation "core_artifact"

  deadlock detected
  DETAIL:  Process 11111 waits for ShareLock on transaction 11111112; blocked by process 11112.
  Process 11112 waits for ShareLock on transaction 11111111; blocked by process 11111.
  HINT:  See server log for query details.
  CONTEXT:  while inserting index tuple (1,2) in relation "core_artifact_sha256_key"

  deadlock detected
  DETAIL:  Process 11114 waits for ShareLock on transaction 11111114; blocked by process 11113.
  Process 11113 waits for ShareLock on transaction 11111113; blocked by process 11114.
  HINT:  See server log for query details.
  CONTEXT:  while rechecking updated tuple (123,45) in relation "core_contentartifact"

  • If the mentioned relation is rpm_package and/or you are on Satellite/Capsule 6.12.1+, then please read this Solution / This content is not included.this BZ and follow this solution for remedy steps (that apply to both types of deadlocks).

  • If the mentioned relation is rpm_modulemd and/or you are on Satellite/Capsule 6.13, follow the resolution as mentioned in this article.

Resolution

A corrective for this is presently being worked on by Red Hat Engineering:

Workaround: 1

  • Stop all workers on the impacted Capsules except 1 and synchronize. Note that this will increase time required for the synchronization to complete.

  • Wait for any currently running Capsule synchronization to finish.

  • On Capsule.

    # systemctl stop pulpcore-worker@{2..32}
# systemctl status pulpcore-worker@{1..32} | grep Active

  • Start new Optimized Capsule Synchronization.

Workaround: 2

  • Instead of Complete Capsule sync initiate optimized Capsule sync to sync only the latest changes.

Note: Optimized Sync only synchronizes the repository bypassing RPMs that have not detected differences from the upstream(Satellite's) RPMs.

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

If the Capsule Sync issue persists after upgrading the Red Hat Satellite server, please open up a support case with This content is not included.Red Hat Technical Support.

Root Cause

  • When syncing multiple repositories with similar contents concurrently, Pulp workers could update the same database record at the same time which could cause deadlock errors.

Diagnostic Steps

  • Check Actions::Katello::CapsuleContent::Sync details and found one or more of the below deadlock errors:

      Task errors: deadlock detected
  DETAIL:  Process 00001 waits for ShareLock on transaction 1000001; blocked by process 00001.
  Process 00001 waits for ShareLock on transaction 1000001; blocked by process 00002.
  HINT:  See server log for query details.
  CONTEXT:  while rechecking updated tuple (0001,1) in relation "core_artifact"

  deadlock detected
  DETAIL:  Process 11111 waits for ShareLock on transaction 11111112; blocked by process 11112.
  Process 11112 waits for ShareLock on transaction 11111111; blocked by process 11111.
  HINT:  See server log for query details.
  CONTEXT:  while inserting index tuple (1,2) in relation "core_artifact_sha256_key"

  deadlock detected
  DETAIL:  Process 11114 waits for ShareLock on transaction 11111114; blocked by process 11113.
  Process 11113 waits for ShareLock on transaction 11111113; blocked by process 11114.
  HINT:  See server log for query details.
  CONTEXT:  while rechecking updated tuple (123,45) in relation "core_contentartifact"

  deadlock detected
  DETAIL: Process 50115 waits for ShareLock on transaction 5207924; blocked by process 50122. 
  Process 50122 waits for ShareLock on transaction 5207925; blocked by process 50115.
  HINT: See server log for query details.
  CONTEXT: while updating tuple (200,3) in relation "rpm_modulemd" Plus 1 more error
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.