Publishing a new version of a Content View in Red Hat Satellite 6 fails with an error duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq"

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • Publishing a new version of a Content View fails with the following traceback:

    Jan 24 15:39:23 nodename pulpcore-worker-6: pulp [7ee38a97-151c-47a7-8739-0ff4e44f2973]: pulpcore.tasking.pulpcore_worker:INFO: Starting task cd1693d0-8a15-4b95-a6f6-c431fb376865
Jan 24 15:39:23 nodename pulpcore-api: pulp [7ee38a97-151c-47a7-8739-0ff4e44f2973]:  - - [24/Jan/2022:20:39:23 +0000] "POST /pulp/api/v3/repositories/rpm/rpm/376c26da-1fd7-41ef-b33e-071908c17d1f/modify/ HTTP/1.1" 202 67 "-" "OpenAPI-Generator/3.13.3/ruby"
Jan 24 15:39:23 nodename pulpcore-worker-8: pulp [7ee38a97-151c-47a7-8739-0ff4e44f2973]: pulpcore.tasking.pulpcore_worker:INFO: Task 3f572446-f9c4-440b-94cb-3bc4f39db952 failed (duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq"
Jan 24 15:39:23 nodename pulpcore-worker-8: DETAIL:  Key (repository_id, content_id, version_removed_id)=(efb0eb39-f02f-4493-a1af-bf6fb2b462de, 2b6f6ef7-3382-4b7c-a8a1-f088db1bb581, 61dfdd1f-46d1-49b7-89bc-44adb55769ec) already exists.

  • Optionally, an attempt to delete a CV or its version fails with the same error.

Resolution

  • The issue is resolved in advisory RHBA-2022:4850.
  • Users of Red Hat Satellite are advised to upgrade to the updated version, which fix these bugs.
  • Please note that the fix prevents the issue. It does not remedy the already broken repository version.
  • For the already affected Content Views the only solution is:
    * Clone the existing content view in the Web UI
    * Publish and promote the cloned Content View
    * Move the hosts to consume the cloned Content View.
    * Remove the affected Content View.

Note: Before applying this update, make sure all previously released errata relevant to your system have been applied, Refer: Updating Satellite Server, Capsule Server, and Content Hosts

Remedy steps during Content View deletion

In case Content View (version) deletion is affected by the problem, then remove the whole Content View using below procedure:

  • From the error string Key (repository_id, content_id, version_removed_id)=(efb0eb39-f02f-4493-a1af-bf6fb2b462de, 2b6f6ef7-3382-4b7c-a8a1-f088db1bb581, 61dfdd1f-46d1-49b7-89bc-44adb55769ec) already exists., copy repository_id (first UUID, here the efb0eb39-f02f-4493-a1af-bf6fb2b462de) and use it to delete the pulp repository:

    repo=efb0eb39-f02f-4493-a1af-bf6fb2b462de
certs="--cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/pki/katello/certs/pulp-client.crt --key /etc/pki/katello/private/pulp-client.key"
hname=$(hostname -f)
curl -X DELETE $certs https://${hname}/pulp/api/v3/repositories/rpm/rpm/${repo}/

    That will spawn a pulp task with href printed to stdout, like:

    {"task":"/pulp/api/v3/tasks/cac29f13-8492-490b-856c-524a1f3ef15f/"}

    Check the task details (use proper href / task id):

    curl -s $certs https://${hname}/pulp/api/v3/tasks/cac29f13-8492-490b-856c-524a1f3ef15f/ | json_reformat
{
    "pulp_href": "/pulp/api/v3/tasks/cac29f13-8492-490b-856c-524a1f3ef15f/",
    "pulp_created": "2022-07-26T11:36:54.103650Z",
    "state": "completed",
..

  • Once the task is completed, delete all CV versions (that should pass properly).

  • Then delete the CV itself - it will fail on step Actions::Pulp3::ContentView::DeleteRepositoryReferences .

  • Open the dynflow console of the task, and click to Skip link to this failed state.

  • Then click to the Resume link on the top

  • Refresh the page until the task completes

Then, the CV will be deleted.

For more KB articles/solutions related to Red Hat Satellite 6.x Content View Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Content View Issues.

Root Cause

  • Due to a bug in Satellite, it loses the tracking of the contents of the Content View version.
  • The error is observed when a package filter is used in the Content View and all of the following conditions are met in the following order:
    1. Some new packages are added and other removed from the filter rule and a new version of the Content View is published.
    2. Some of the previously removed packages are re-added to the filter and a new version of the Content View is published.
    3. Satellite runs its scheduled remove orphans task or the task is run manually by the user.
    4. Some of the previously re-added packages are again removed from the filter and an attempt is made to publish a new version of the Content View, which now triggers the bug and fails.
  • Due to this bug, duplicate IDs for the Content View are observed in the database of the pulpcore component, which is referred by Satellite for the repository management.

Diagnostic Steps

Steps to reproduce

  • Create a content view and attach "Red Hat Satellite Tools 6.10 for RHEL 7 Server RPMs x86_64" repo.

  • Create a package include filter, In the package filter:
    * Add 2 rules to include 'foreman-cli' and 'katello-agent' rpms.

  • Publish the content view. Content View version 1.0 should have the following 2 rpms:

  • foreman-cli
  • katello-agent

  • Amend the package filter:

    • Add 1 rule to include 'katello-host-tools' rpm.

  • Publish the content view again. Content View version 2.0 should have the following 3 rpms:

  • foreman-cli
  • katello-agent
  • katello-host-tools

  • Amend the package filter:

    • Remove 2 rules that included 'foreman-cli' and 'katello-agent'.
    • Add 1 rule to include 'katello-host-tools-fact-plugin' rpm.

  • Publish the content view again. Content View version 3.0 should have the following 2 rpms:

    • katello-host-tools
    • katello-host-tools-fact-plugin

  • Amend the package filter:

    • Add 2 rules to include 'foreman-cli' and 'katello-agent' rpms.

  • Publish the content view again. Content View version 4.0 should have the following 4 rpms:

  • foreman-cli
  • katello-agent
  • katello-host-tools
  • katello-host-tools-fact-plugin

  • Manually trigger a clean orphan task. Wait until the task is finished in Web UI -> Monitor -> Task page.

    foreman-rake katello:delete_orphaned_content

  • Amend the package filter:

    • Remove 2 rules that included 'foreman-cli' and 'katello-agent'.

  • Publish the content view again.

  • In /var/log/messages file, the following Pulp task error is observed.

     # tail -f /var/log/messages
pulpcore.tasking.pulpcore_worker:INFO: Task d325195f-1ab8-4699-a6b2-11218c86686b failed (duplicate key value violates unique constraint "core_repositorycontent_repository_id_content_id_df902e11_uniq"
pulpcore-worker-3: DETAIL:  Key (repository_id, content_id, version_removed_id)=(73390261-c8e8-4f35-af62-e06b186ba7d1, 829b918a-f070-48d4-af09-f688f9f9e4c2, cbcf1dc4-fa8f-46b9-8348-d432ecde3958) already exists.
pulpcore-worker-3: )
pulpcore-worker-3: pulp [4bfaf4e2-64c9-4277-b2dd-8d2b7b303511]: pulpcore.tasking.pulpcore_worker:INFO:   File "/usr/lib/python3.6/site-packages/pulpcore/tasking/pulpcore_worker.py", line 317, in _perform_task
pulpcore-worker-3: result = func(*args, **kwargs)
pulpcore-worker-3: File "/usr/lib/python3.6/site-packages/pulpcore/app/tasks/repository.py", line 218, in 
add_and_remove
<snip>

Foreman task didn't raise error but content view still containing the same packages as the previous version. Excluded rpms are not removed.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.