Capsule sync fails with "duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_XXXXX_uniq" error

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.10 and 6.11

Issue

  • Getting following error when synchronizing Capsule.

        duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_xxxxx_uniq"
    DETAIL:  Key (repository_id, number)=(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, 1) already exists.
    Could not lookup a publication_href for repo 176
    
    insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_xxxxx_fk_core_repo"
    DETAIL:  Key (version_added_id)=(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) is not present in table "core_repositoryversion".
    Could not lookup a publication_href for repo 181

    insert or update on table "core_repositorycontent" violates foreign key constraint "core_repositoryconte_version_added_id_xxxxx_fk_core_repo"
    DETAIL:  Key (version_added_id)=(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) is not present in table "core_repositoryversion".
    Could not lookup a publication_href for repo 184

Resolution

  • Solution 1: Upgrade the Satellite and Capsule servers to 6.12 or newer version.

  • Solution 2:: Hotfix is available for Satellite 6.11.5. Follow the steps in the bugzilla links below to install the hotfix.
    NOTE: This hotfix needs to be deployed on Capsule servers to prevent the issue of failing Capsule syncs when syncing multiple kickstart repositories:

  • Solution 3: If the solution above couldn't solve the issue. The bug could be causing the data corruption.

    • WARNING: Make a VM snapshot of the affected Capsule before proceeding.

    • Get the repository id from the below error message.

    duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_xxxxx_uniq"
DETAIL:  Key (repository_id, number)=(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, 1) already exists.
    • In the affected Capsule, run the following SQL query to get the latest_version of the repository.
    su - postgres -c "psql pulpcore -c \"select max(number) from core_repositoryversion where repository_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'\""
    • Run the following SQL query to get the next_version of the repository.
    su - postgres -c "psql pulpcore -c \"select name, next_version from core_repository where pulp_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'\""
    • If the latest_version is equal to the next_version, then we will need to manually increase the next_version by 1. For example, if both latest_version and next_version is 3, then we need to set the next_version to 4.
        # Example:
su - postgres -c "psql pulpcore -c \"update core_repository set next_version = 4 where pulp_id = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'\""
    • Trigger an Optimized Capsule Synchronization

Root Cause

  • Pulp stores only one copy of the sub repository per treeinfo checksum so multiple same kickstart repositories share the same sub repositories. When syncing those repositories concurrently, race conditions occur trying to update the same sub-repositories.
  • This bug This content is not included.Bug 2077363 has been closed with a resolution of ERRATA RHSA-2023:2097.

Diagnostic Steps

  • Check Actions::Katello::CapsuleContent::Sync details and found the following error:
  error:
    traceback: |2
        File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 339, in _perform_task
          result = func(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 474, in synchronize
          subrepo_version = dv.create()
        File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 145, in create
          with self.repository.new_version() as new_version:
        File "/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py", line 258, in new_version
          version.save()
        File "/usr/lib/python3.6/site-packages/django_lifecycle/mixins.py", line 134, in save
          save(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 744, in save
          force_update=force_update, update_fields=update_fields)
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 782, in save_base
          force_update, using, update_fields,
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 873, in _save_table
          result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
        File "/usr/lib/python3.6/site-packages/django/db/models/base.py", line 911, in _do_insert
          using=using, raw=raw)
        File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method
          return getattr(self.get_queryset(), name)(*args, **kwargs)
        File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 1186, in _insert
          return query.get_compiler(using=using).execute_sql(return_id)
        File "/usr/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1377, in execute_sql
          cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 67, in execute
          return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 76, in _execute_with_wrappers
          return executor(sql, params, many, context)
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
        File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 89, in __exit__
          raise dj_exc_value.with_traceback(traceback) from exc_value
        File "/usr/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
          return self.cursor.execute(sql, params)
    description: |
      duplicate key value violates unique constraint "core_repositoryversion_repository_id_number_xxxxx_uniq"
      DETAIL:  Key (repository_id, number)=(xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx, 1) already exists.
  worker: "/pulp/api/v3/workers/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/"
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.