[Satellite 6.2] pulp raises "DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_erratum"

Environment

Red Hat Satellite 6

Issue

• upgrading Satellite from 6.1 to 6.2 or from 6.2 to 6.3
• the upgrade failed or post upgrade pulp raises error: "E11000 duplicate key error index: pulp_database.units_erratum..."

Resolution

Remove the erroneusly created duplicates in mongo database and update references to them to the remaining representative record.

Warning: below steps directly modify pulp/mongo database, if uncertain doing so, consult the steps with Red Hat support first

• stop pulp and take mongo backup:

for i in pulp_resource_manager pulp_workers pulp_celerybeat mongod; do service $i stop; done

tar --selinux --create --gzip --file=mongo_data.tar.gz --listed-incremental=mongo.snar --exclude=mongod.lock /var/lib/mongodb/

service mongod start

• run attacher script:

# ./remove_dups_units_erratum.sh 
Fri Sep  9 14:33:59 CEST 2016: Detecting duplicates in given collection..
Fri Sep  9 14:34:02 CEST 2016: Preparing commands to remove the duplicates..
Fri Sep  9 14:34:04 CEST 2016: removing the duplicates from mongo (39 mongo commands)..
Fri Sep  9 14:34:05 CEST 2016: detecting potential duplicates in repo_content_unit (can take several minutes)..
Fri Sep  9 14:34:30 CEST 2016: removing duplicates in repo_content_unit (0 mongo commands)..
Fri Sep  9 14:34:30 CEST 2016: finished. All 39 changes written to "work_done.js" for audit/review.
#

• optionally, review the changes done as written to work_done.js file

• run pulp-manage-db command to ensure the unique index is created and verified against the collection:

sudo -u apache pulp-manage-db

• start pulp services:

for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i start; done

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

Original root cause behind presence of the duplicates is unknown. The attached script fixes the duplicates by:

• identifying what units_erratum entries are duplicated
• removing the duplicates
• in some other collections, replacing references to duplicate entries by the remaining/original record
• identifying and removing potential duplicates created in previous step in the other collections (namely in repo_content_unit, to prevent a repository has associated the same unit twice)

Diagnostic Steps

Pulp raising error:

DuplicateKeyError: E11000 duplicate key error index: ..

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.