[Sat6.3] - Upgrade to Sat 6.3.5 stucked in Upgrade Step: add_export_distributor "Errno::ENOENT: No such file or directory @ rb_sysopen"

Solution Verified - Updated

Environment

  • Satellite 6.3

Issue

  • Upgrade stucked on Upgrade Step: add_export_distributor:
Upgrade Step: add_export_distributor (this may take a while) ...
[ INFO 2018-12-20 15:40:50 verbose] Upgrade Step: add_export_distributor (this may take a while) ...
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
rake aborted!
ForemanTasks::TaskError: Task 1a584ee8-594d-4bcb-a92a-3d2db3a6c954: Errno::ENOENT: No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/462/C60/.permissions_check.46875280.18905.663078; Errno::ENOENT: No such file or directory @ rb_sysopen - 
:
[ERROR 2018-12-20 16:00:16 verbose] Upgrade step add_export_distributor failed.

Resolution

  • Clean the foreman cachec /usr/share/foreman/tmp/cache
# foreman-rake tmp:cache:clear

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Root Cause

Foreman cache /usr/share/foreman/tmp/cache is filled up

Diagnostic Steps

  • Resume any task that causes the stucked in satellite upgrade:
  • RefreshRepository, example:
Id: 125af7c6-ce5e-4c5a-99ad-c48bd024d7a7   
Label: Actions::Katello::Repository::RefreshRepository
Status: paused Resume
  • CapsuleContent sync cleanup, example:
# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::CapsuleContent::Sync' STATES='paused' VERBOSE=true
# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::CapsuleContent::Sync' RESULT='error' VERBOSE=true
# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label = Actions::Katello::CapsuleContent::Sync' RESULT='pending' VERBOSE=true
  • Check the installer script:
   # satellite-installer --scenario satellite --upgrade --verbose --noop

then run installer as the output has no error:

   # satellite-installer --scenario satellite --upgrade --verbose

Result is getting the same issue with a different task id this time:

Upgrade Step: add_export_distributor (this may take a while) ...
[ INFO 2018-12-20 15:40:50 verbose] Upgrade Step: add_export_distributor (this may take a while) ...
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
rake aborted!
ForemanTasks::TaskError: Task 1a584ee8-594d-4bcb-a92a-3d2db3a6c954: Errno::ENOENT: No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/462/C60/.permissions_check.46875280.18905.663078; Errno::ENOENT: No such file or directory @ rb_sysopen - 
:
[ERROR 2018-12-20 16:00:16 verbose] Upgrade step add_export_distributor failed.

  • Clean existing tasks.

    • did resume this RefreshRepository task and make sure no other tasks are paused/error/pending.

  • Clear the foreman cache and check the permission of foreman temp cache folder; /usr/share/foreman/tmp/cache

# ls -lrt /usr/share/foreman/tmp/cache
rwxr-xr-x. 3 foreman foreman 60 Dec 20 14:30 462
drwxr-xr-x. 3 foreman foreman 60 Dec 20 14:30 46A
drwxr-xr-x. 3 foreman foreman 60 Dec 20 14:30 5BF

# foreman-rake tmp:cache:clear
  • Attempt the upgrade script:
 # satellite-installer --scenario satellite --upgrade --verbose
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.