Capsule is not able to sync

Solution Verified - Updated 14 Jun 2024

Environment

Satellite 6.4
Capsule 6.4

Issue

One of the capsules is not completing the sync task.
The sync started three days ago and 11% complete.

# hammer capsule content synchronization-status --id 16
Last sync:                    2019/04/24 19:50:16
Status:                       3 environment(s) can be synchronized: Development, Front Runner, Production
Currently running sync tasks:
 1) Task id:  8361f6c3-9ae9-408f-a7ad-30c47598449b
    Progress: 11%
Last failure:
    Task id:  ab5355f8-5b9e-4310-9a29-76cacd7d54c9
    Messages:
      500 Internal Server Error
      500 Internal Server Error
      500 Internal Server Error
      500 Internal Server Error
      500 Internal Server Error
      500 Internal Server Error
      500 Internal Server Error
      500 Internal Server Error
      500 Internal Server Error

Resolution

Correct the ownership of /var/lib/mongodb

# chown -R mongodb.mongodb /var/lib/mongodb

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

Root Cause

The ownership on /var/lib/mongodb/ is not correct
Files should be owned by user mongodb and group mongodb

Diagnostic Steps

The messages reports failure to start mongodb

# tail var/log/messages 
May 16 14:36:12 capsule.example.com: kombu.async.hub:ERROR: (160006-05344)     worker_watcher.handle_worker_heartbeat(name)
May 16 14:36:12 capsule-example.com: kombu.async.hub:ERROR: (160006-05344)   File "/usr/lib/python2.7/site-packages/pulp/server/async/worker_watcher.py", line 51, in handle_worker_heartbeat
May 16 14:36:12 capsule-example.com: kombu.async.hub:ERROR: (160006-05344)     upsert=True)
May 16 14:36:12 capsule-example.com: kombu.async.hub:ERROR: (160006-05344)   File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 516, in update_one
May 16 14:36:12 capsule-example.com: kombu.async.hub:ERROR: (160006-05344)     upsert=upsert, multi=False, write_concern=write_concern, **update)
May 16 14:36:12 capsule-example.com: kombu.async.hub:ERROR: (160006-05344)   File "/usr/lib/python2.7/site-packages/pulp/server/db/querysets.py", line 100, in update
May 16 14:36:12 capsule-example.com: kombu.async.hub:ERROR: (160006-05344)     super(CriteriaQuerySet, self).update(*args, **kwargs)
May 16 14:36:12 capsule-example.com: kombu.async.hub:ERROR: (160006-05344)   File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 472, in update
May 16 14:36:12 capsule-example.com: kombu.async.hub:ERROR: (160006-05344)     raise OperationError(u'Update failed (%s)' % unicode(err))
May 16 14:36:12 capsule-example.com: kombu.async.hub:ERROR: (160006-05344) OperationError: Update failed (command SON([('update', u'workers'), ('ordered', True), ('updates', [SON([('q', {'_id': 'reserved_resource_worker-0@capsule-hou6-hpc'}), ('u', {'$set': {'last_heartbeat': datetime.datetime(2019, 5, 16, 19, 36, 12, 793301)}}), ('multi', False), ('upsert', True)])])]) on namespace pulp_database.$cmd failed: exception: Can't take a write lock while out of disk space)

Mongodb log reports Permission denied

# tail  var/log/mongodb/mongodb.log
2019-05-16T14:36:01.715-0500 [FileAllocator] error: failed to allocate new file: /var/lib/mongodb/pulp_database.11 size: 2146435072 boost::filesystem::create_directory: Permission denied: "/var/lib/mongodb/_tmp".  will try again in 10 seconds
2019-05-16T14:36:11.715-0500 [FileAllocator] allocating new datafile /var/lib/mongodb/pulp_database.11, filling with zeroes...
2019-05-16T14:36:11.716-0500 [FileAllocator] creating directory /var/lib/mongodb/_tmp
2019-05-16T14:36:11.716-0500 [FileAllocator] error: failed to allocate new file: /var/lib/mongodb/pulp_database.11 size: 2146435072 boost::filesystem::create_directory: Permission denied: "/var/lib/mongodb/_tmp".  will try again in 10 seconds
2019-05-16T14:36:21.716-0500 [FileAllocator] allocating new datafile /var/lib/mongodb/pulp_database.11, filling with zeroes...
2019-05-16T14:36:21.717-0500 [FileAllocator] creating directory /var/lib/mongodb/_tmp
2019-05-16T14:36:21.717-0500 [FileAllocator] error: failed to allocate new file: /var/lib/mongodb/pulp_database.11 size: 2146435072 boost::filesystem::create_directory: Permission denied: "/var/lib/mongodb/_tmp".  will try again in 10 seconds
2019-05-16T14:36:31.717-0500 [FileAllocator] allocating new datafile /var/lib/mongodb/pulp_database.11, filling with zeroes...
2019-05-16T14:36:31.718-0500 [FileAllocator] creating directory /var/lib/mongodb/_tmp
2019-05-16T14:36:31.718-0500 [FileAllocator] error: failed to allocate new file: /var/lib/mongodb/pulp_database.11 size: 2146435072 boost::filesystem::create_directory: Permission denied: "/var/lib/mongodb/_tmp".  will try again in 10 seconds

Check ownership of directory

# ls -dltra  /var/lib/mongodb/
drwxr-xr-x. 3 root root 4096 Apr 10 13:22 /var/lib/mongodb/

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.