Capsule synchronization seems stuck at 10% or 11% for a long time

Solution Verified - Updated

Environment

  • Red Hat Satellite 6
    • an external Capsule

Issue

Invoking a Capsule Sync:

  • it reaches progress of 10% or 11% quite soon
  • but then it is stuck there for a long time
  • after a long time (i.e. half of the whole Caps sync), the progress bar starts to move again

Is that normal?
Isn't the Capsule synchronization really stuck on the background at the 10% or 11% of progress?

Resolution

If pulp tasks on Capsule are progressing (see Diagnostic Steps for details), then "sync stuck at 10%" is normal and expected behaviour. See Root Cause for explanation.

An improvement in this behaviour has been requested in This content is not included.this bugzilla.

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

For more KB articles/solutions related to Red Hat Satellite 6.x Pulp 2.0 Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Pulp 2.0-related Issues

Root Cause

The progress bar being stuck at 10% is caused by the specific way of counting the progress in foreman tasks versus how the real progress happens in pulp on the Capsule. An elevator-pitch-like description follows.

From foreman tasks perspective, Capsule Sync consists of some auxiliary and generic steps, followed by one step for each repo, to sync+publish it. At the time of completing the auxiliary&generic steps and spawning the "one step per repo" steps, the whole task reaches the 10% or 11% progress. And here it waits for the first "repo sync step" to be completed.

From pulp perspective, however, the Capsule first gets a bulk of requests to sync a set of repos, that automatically generate one publish task for each repo. That publish task is enqueued to the end of the work queue on the Capsule. But the "foreman's step for repo sync" is marked as completed only after this publish task is completed. From another perspective, Capsule simply starts with half of the work on each and every foreman's step, before it completes the first one.

On some particular example: assume a Capsule Sync consists of syncing 5 repos R1 to R5. That means 5 dynflow steps are run concurrently, which spawn pulp tasks R1sync+R1publish, R2sync+R2publish, etc. Capsule gets and executes tasks in ordering R1sync, R2sync,.. R5sync, R1publish, .. R5publish. Therefore, with some level of abstraction, there is a time when all repos content is synced but has not yet published anything, whole Capsule sync is roughly at one half, BUT neither dynflow step is yet completed - hence the task itself still sits at 10%!

Diagnostic Steps

To ensure Capsule is not stuck with pulp tasks, run repeatedly:

pulp-admin -u admin -p $(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2) tasks list

or optionally pulp API directly:

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)
curl -ks -u admin:$pulpAdminPassword -X POST -d '{ "criteria": { "filters": { "state": { "$in": [ "waiting", "running" ] }}}}' https://$(hostname -f)/pulp/api/v2/tasks/search/ | python -m json.tool

Redirect the output to a file or grep for numbers of pending and running tasks. If those numbers evolve in time, the Capsule sync is in progress.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.