Pulp tasks are not processed in Red Hat Satellite and it hangs in the running state with error E11000 duplicate key error index: pulp_database.task_status.$task_id_-1 dup key

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • Pulp tasks hangs in running state and are not processed with below error observed in /var/log/messages:

    INTERNAL ERROR: NotUniqueError(u'Tried to save duplicate unique keys (insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.task_status.$task_id_-1  dup key: { : "xxxx" }

Resolution

  • Uninstall the mongoengine package installed trough pip.

    # pip uninstall mongoengine

  • Reinstall the mongoengine package from rhel-7-server-satellite-6.x-rpms repository and restart the Satellite services.

        # yum reinstall python-mongoengine

   # katello-service restart

  • If issue still persist with mentioned error observed in /var/log/messages, upgrade Red Hat Satellite to version 6.6.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

  • The mentioned error is observed in two scenarios.

Diagnostic Steps

  • If the version of mongoengine package obtained from the output of below commands are different, it implies that mongoengine is installed using pip.

       # pip freeze | grep mongoengine
   DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A future version of pip will drop support for Python 2.7.
   mongoengine==0.16.3

   # rpm -qa | grep mongoengine
   python-mongoengine-0.10.5-2.el7sat.noarch

  • Below error is observed in /var/log/messages:

       Feb  7 12:00:38 satellite pulp: celery.worker.job:CRITICAL: (86093-53696) Task pulp.server.managers.consumer.applicability.regenerate_applicability_for_consumers[22320cf3-b12d-49da-b76f-96fa6a2b5707] INTERNAL ERROR: NotUniqueError(u'Tried to save duplicate unique keys (insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.task_status.$task_id_-1  dup key: { : "22320ab9-b12d-49da-g45f-12fa6a2b1234" })',)
   Feb  7 12:00:38 satellite pulp: celery.worker.job:CRITICAL: (86093-53696) NotUniqueError: Tried to save duplicate unique keys (insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.task_status.$task_id_-1  dup key: { : "22320ab9-b12d-49da-g45f-12fa6a2b1234" })
   Feb  7 13:53:38 satellite pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[7ce12345-f7c0-4f89-a156-98765a556fa0] succeeded in 0.0112077174708s: None
Feb  7 12:00:38 satellite pulp: py.warnings:WARNING: (17054-53696) /usr/lib64/python2.7/site-packages/pymongo/topology.py:149: UserWarning: MongoClient opened before fork. Create MongoClient only after forking. See PyMongo's documentation for details: http://api.mongodb.org/python/current/faq.html#is-pymongo-fork-safe
   Feb  7 12:00:38 satellite pulp: py.warnings:WARNING: (17054-53696)   "MongoClient opened before fork. Create MongoClient only "
   Feb  7 12:00:38 satellite pulp: py.warnings:WARNING: (17054-53696)
   Feb  7 12:00:38 satellite pulp: py.warnings:WARNING: (17054-53696) /usr/lib/python2.7/site-packages/celery/app/trace.py:364: RuntimeWarning: Exception raised outside body: NotUniqueError(u'Tried to save duplicate unique keys (insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.task_status.$task_id_-1  dup key: { : "f5b12345-f870-4af4-ba3c-195e2e658c19" })',):
   Feb  7 12:00:38 satellite pulp: py.warnings:WARNING: (17054-53696) NotUniqueError: Tried to save duplicate unique keys (insertDocument :: caused by :: 11000 E11000 duplicate key error index: pulp_database.task_status.$task_id_-1  dup key: { : "f5b08289-f870-4af4-ba3c-195e2e658c19" })
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.