[Satellite 6.2] pulp raises DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_docker_tag.$name_1_repo_id_1

Environment

• Red Hat Satellite 6.2
  • being upgraded from 6.1.9

Issue

• upgrading Satellite 6.1.9 to 6.2
  • manually running pulp processes during the upgrade (not recommended)
• the upgrade failed or post upgrade pulp raises error: "DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_docker_tag.$name_1_repo_id_1 dup key: { : "6.7", : "our-containers-rhel" }

Resolution

Remove the erroneusly created duplicates in mongo database and update references to them to the remaining representative record.

Warning: below steps ask you to modify database directly, if uncertain doing so, consult the steps with Red Hat support first

• stop pulp and take mongo backup:

for i in pulp_resource_manager pulp_workers pulp_celerybeat mongod; do service $i stop; done

tar --selinux --create --gzip --file=mongo_data.tar.gz --listed-incremental=mongo.snar --exclude=mongod.lock /var/lib/mongodb/

service mongod start

• run attached script:

# ./remove_dups_units_docker_tag.sh
Fri Sep  9 15:05:48 CEST 2016: Detecting duplicates in units_docker_tag collection..
Fri Sep  9 15:05:48 CEST 2016: Preparing commands to remove the duplicates..
Fri Sep  9 15:05:48 CEST 2016: removing the duplicates from mongo (230 mongo commands)..
Fri Sep  9 15:05:49 CEST 2016: detecting potential duplicates in repo_content_unit (can take several minutes)..
Fri Sep  9 15:06:50 CEST 2016: removing duplicates in repo_content_unit (0 mongo commands)..
Fri Sep  9 15:06:50 CEST 2016: finished. All 230 changes written to "work_done.js" for audit/review.
#

• optionally, review the changes done as written to work_done.js file

• run pulp-manage-db command to ensure the unique index is created and verified against the collection:

sudo -u apache pulp-manage-db

• start pulp services:

for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i start; done

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

pulp-manage-db running during the upgrade needs pulp services are stopped. Otherwise, parallel upgrading the DB and writing to it breaks some unique keys on some collections/tables.

The attached script fixes the duplicates by:

• identifying what units_docker_tag entries are duplicated
• removing the duplicates
• in some other collections, replacing references to duplicate entries by the remaining/original record
• identifying and removing potential duplicates created in previous step in the other collections (namely in repo_content_unit, to prevent a repository has associated the same unit twice)

Diagnostic Steps

Pulp raising error:

DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_docker_tag.$name_1_repo_id_1  dup key: { : "6.7", : "our-containers-rhel" }

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.