[Satellite 6.2] pulp raises E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.2.x
  • being upgraded from 6.1.9
  • Red hat Satellite 6.3

Issue

  • Synchronizing OSTree / atomic hosts to Satellite
  • Upgrading to Satellite 6.2
  • Under various scenarios, pulp raises error E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1
  • While upgrading the capsule

Resolution

Remove the erroneusly created duplicates in mongo database and update references to them to the remaining representative record.

Warning: below steps ask you to modify database directly, if uncertain doing so, consult the steps with Red Hat support first

  • stop pulp and take mongo backup:
for i in pulp_resource_manager pulp_workers pulp_celerybeat mongod; do service $i stop; done

tar --selinux --create --gzip --file=mongo_data.tar.gz --listed-incremental=mongo.snar --exclude=mongod.lock /var/lib/mongodb/

service mongod start
  • run attached script:
# ./remove_dups_units_ostree.sh
Fri Sep  9 15:33:57 CEST 2016: Detecting duplicates in units_ostree collection..
Fri Sep  9 15:33:57 CEST 2016: Preparing commands to remove the duplicates..
Fri Sep  9 15:33:57 CEST 2016: removing the duplicates from mongo (10 mongo commands)..
Fri Sep  9 15:33:57 CEST 2016: detecting potential duplicates in repo_content_unit (can take several minutes)..
Fri Sep  9 15:33:59 CEST 2016: removing duplicates in repo_content_unit (0 mongo commands)..
Fri Sep  9 15:33:59 CEST 2016: finished. All 10 changes written to "work_done.js" for audit/review.
#
  • run pulp-manage-db command to ensure the unique index is created and verified against the collection:
sudo -u apache pulp-manage-db
  • start pulp services:
for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i start; done

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

Installation of or upgrade to Satellite 6.2 did not apply unique keys into several mongo collections/tables due to an error, causing duplicate entries to appear. In our case, units_ostree mongo collection needs to have entries with unique tripple "(remote_id;branch;commit)".

The goal of steps in Resolution is to remove duplicated entries / OSTree units and replace references to the duplicates to the remaining unique OSTree unit. The attached script fixes the duplicates by:

  • identifying what units_ostree entries are duplicated
  • removing the duplicates
  • in some other collections, replacing references to duplicate entries by the remaining/original record
  • identifying and removing potential duplicates created in previous step in the other collections (namely in repo_content_unit, to prevent a repository has associated the same unit twice)

Diagnostic Steps

  • under various scenarios (including pulp-manage-db, below error with that backtrace is raised:
E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1  dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" }
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 193, in main
    return _auto_manage_db(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 256, in _auto_manage_db
    migrate_database(options)
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 142, in migrate_database
    ensure_database_indexes()
  File "/usr/lib/python2.7/site-packages/pulp/server/db/manage.py", line 179, in ensure_database_indexes
    model_class.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 738, in ensure_indexes
    collection = cls._get_collection()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 210, in _get_collection
    cls.ensure_indexes()
  File "/usr/lib/python2.7/site-packages/mongoengine/document.py", line 766, in ensure_indexes
    collection.create_index(fields, background=background, **opts)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1380, in create_index
    self.__create_index(keys, kwargs)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 1290, in __create_index
    sock_info, cmd, read_preference=ReadPreference.PRIMARY)
  File "/usr/lib64/python2.7/site-packages/pymongo/collection.py", line 205, in _command
    read_concern=read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/pool.py", line 211, in command
    read_concern)
  File "/usr/lib64/python2.7/site-packages/pymongo/network.py", line 100, in command
    helpers._check_command_response(response_doc, msg, allowable_errors)
  File "/usr/lib64/python2.7/site-packages/pymongo/helpers.py", line 189, in _check_command_response
    raise DuplicateKeyError(errmsg, code, response)
DuplicateKeyError: E11000 duplicate key error index: pulp_database.units_ostree.$remote_id_1_branch_1_commit_1  dup key: { : "174b6356ebe246768ac1600709cfefc6589a43503397a9e7589111d4da9188f6", : "rhel-atomic-host/7/x86_64/standard", : "b672bf8a457cb28e003dee20c53749636ef5fce3e4743afe4aaad269d3aaa62a" }
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.