[Satellite6] Synchronizing Atomic OSTRee repo fails, pulp worker dies

Environment

• Red Hat Satellite 6
• ostree-2016.5 or older

Issue

• trying to synchronize an Atomic OSTRee repo fails everytime
• checking /var/log/messages, a pulp worker died that time

Resolution

Update ostree package to ostree-2016.15, ostree-2017.1 or anything newer, and apply the fix by restarting katello-service:

yum update ostree
katello-service restart

And then try to sync the repo again.

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

There is a This content is not included.bug in ostree causing pulp celery segmentation fault whenever pulp tries to synchronize an OSTree repository. ostree since versions 2016.15 fixes the Content from github.com is not included.upstream bug for the same.

Diagnostic Steps

• /var/log/messages contains:

Feb 22 10:11:39 satellite pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[11a1a4e9-58c3-49e9-92d3-54d8187ad24f]
..
Feb 22 10:11:49 satellite abrt-server: Compressing data
Feb 22 10:11:50 satellite pulp: celery.worker.job:ERROR: (48264-83360) Task pulp.server.managers.repo.sync.sync[11a1a4e9-58c3-49e9-92d3-54d8187ad24f] raised unexpected: WorkerLostError('Worker exited prematurely: signal 6 (SIGIOT).',)
Feb 22 10:11:50 satellite pulp: celery.worker.job:ERROR: (48264-83360) Traceback (most recent call last):
Feb 22 10:11:50 satellite pulp: celery.worker.job:ERROR: (48264-83360)   File "/usr/lib64/python2.7/site-packages/billiard/pool.py", line 1169, in mark_as_worker_lost
Feb 22 10:11:50 satellite pulp: celery.worker.job:ERROR: (48264-83360)     human_status(exitcode)),
Feb 22 10:11:50 satellite pulp: celery.worker.job:ERROR: (48264-83360) WorkerLostError: Worker exited prematurely: signal 6 (SIGIOT).

• checking coredump in generated ABRT report shows backtrace:

#0  0x00007fa3e98ab5f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	  return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
(gdb) bt full
#0  0x00007fa3e98ab5f7 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
        resultvar = 0
        pid = 3646
        selftid = 62380
#1  0x00007fa3e98acce8 in __GI_abort () at abort.c:90
        save_stage = 2
        act = {__sigaction_handler = {sa_handler = 0x7fa3b9ffa95f, sa_sigaction = 0x7fa3b9ffa95f}, sa_mask = {__val = {173, 140338056396801, 173, 140338056396924, 0, 72057594054705152, 12393906174523604992, 3288334336, 140341176936976, 
              140340606541600, 140340606533664, 173, 140340606541600, 140341631038304, 173, 140341483218264}}, sa_flags = -868142155, sa_restorer = 0x3e911b0}
        sigs = {__val = {32, 0 <repeats 15 times>}}
#2  0x00007fa3d4e436d5 in g_assertion_message (domain=domain@entry=0x7fa3cc412558 "OSTree", file=file@entry=0x7fa3cc4133b5 "src/libostree/ostree-repo.c", line=line@entry=3017, 
    func=func@entry=0x7fa3cc4141d0 <__FUNCTION__.33509> "_ostree_repo_read_bare_fd", message=message@entry=0x7fa398001eb0 "assertion failed: (self->mode == OSTREE_REPO_MODE_BARE || self->mode == OSTREE_REPO_MODE_BARE_USER)")
    at gtestutils.c:2292
        lstr = "3017\000\177\000\000\000\b\260\207ڿ\no\221\272\372\001\000\000\000\000\260=Ạ\177\000"
        s = 0x7fa398001f20 "\220 "
#3  0x00007fa3d4e4376a in g_assertion_message_expr (domain=domain@entry=0x7fa3cc412558 "OSTree", file=file@entry=0x7fa3cc4133b5 "src/libostree/ostree-repo.c", line=line@entry=3017, 
    func=func@entry=0x7fa3cc4141d0 <__FUNCTION__.33509> "_ostree_repo_read_bare_fd", expr=expr@entry=0x7fa3cc413db0 "self->mode == OSTREE_REPO_MODE_BARE || self->mode == OSTREE_REPO_MODE_BARE_USER") at gtestutils.c:2307
        s = 0x7fa398001eb0 "assertion failed: (self->mode == OSTREE_REPO_MODE_BARE || self->mode == OSTREE_REPO_MODE_BARE_USER)"
#4  0x00007fa3cc3d10ba in _ostree_repo_read_bare_fd (self=self@entry=0x3e911b0, checksum=<optimized out>, out_fd=out_fd@entry=0x7fa3b9ffadc0, cancellable=cancellable@entry=0x0, error=error@entry=0x7fa3b9ffae60)
    at src/libostree/ostree-repo.c:3016
..

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.