Execute upgrade:run rake task' step is still running on Satellite server even after successful upgrade

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.14
  • Red Hat Satellite 6.16

Issue

  • After upgrade Satellite from 6.14 to 6.15 everything works fine from Satellite point of view but rake process get stuck. Check Diagnostic Steps to verify
  • During Satellite upgrade from 6.16 to 6.17, the upgrade got stuck in the Execute upgrade:run rake task: step towards the end of the process, whereas /var/log/foreman-installer/satellite.log states that the upgrade was successful.

Resolution

  • This issue is tracked in This content is not included.SAT-32143 as well as This content is not included.SAT-37218, and Red Hat Engineering team is working on it.

  • SOLUTION 1 (RECOMMENDED)

    • Rake process take a lot of time to execute katello:correct_repositories at the backend for a satellite having too many repos and CVs, so wait until process terminate.

  • SOLUTION 2 (WORKAROUND)

    • Execute the command from Satellite Server, to check which list of tasks that rake is waiting to be executed

       echo "UpgradeTask.needing_run.all" | foreman-rake console

    • Example Output :

          Loading production environment (Rails 6.1.7.6)
    Switch to inspect mode.
    UpgradeTask.needing_run.all
    #<ActiveRecord::Relation [#<UpgradeTask id: 1, name: "katello:correct_repositories", task_name: 
    "katello:correct_repositories", long_running: true, always_run: true, skip_failure: true, last_run_time: 
    "2024-10-21 11:40:58.909287000 +0000", ordering: 100, subject: "katello", created_at: "2023-09-25 
    09:19:53.771992000 +0000", updated_at: "2024-10-21 11:40:58.911948000 +0000">, #<UpgradeTask id: 2, name: 
    "katello:clean_backend_objects", task_name: "katello:clean_backend_objects", long_running: true, always_run: 
    true, skip_failure: true, last_run_time: "2024-10-21 11:41:45.540381000 +0000", ordering: 100, subject: 
    "katello", created_at: "2023-09-25 09:19:53.774780000 +0000", updated_at: "2024-10-21 11:41:45.542870000 
+0000">]>

    • From Output it's possible to see that rake process is stuck on katello:correct_repositories and katello:clean_backend_objects commands.

    • Terminate satellite-maintain upgrade process 

      # ps -faux | grep "satellite-maintain upgrade" <==== get PID process
# kill -9 <PID>

    • Force running manually

      # foreman-rake katello:correct_repositories
# foreman-rake katello:clean_backend_objects

    • If the output from above command not show any error, so execute command with COMMIT=TRUE added

      # foreman-rake katello:correct_repositories COMMIT=true
# foreman-rake katello:clean_backend_objects COMMIT=true

Reach out to the This content is not included.Red Hat Technical Support in case any further assistance would be required.

For more KB articles/solutions related to Red Hat Satellite 6.x Installation/Upgrade/Update Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Installation/Upgrade/Update Issues.

Diagnostic Steps

  • Verify that var/log/foreman-maintain/foreman-maintain.log show message as below, without specify that process is terminated.

    I, [2024-11-26 12:56:46+0100 #17390]  INFO -- : foreman-maintain command finished with 
I, [2024-11-26 12:56:46+0100 #9997]  INFO -- : --- Execution step 'Procedures::installer::Upgrade' finished ---
I, [2024-11-26 12:56:46+0100 #9997]  INFO -- : --- Execution step 'Execute upgrade:run rake task' [installer-upgrade-rake-task] started --

  • Verify if rake process is still in progress

    # ps ax | grep rake
17478 pts/3    S+     0:00 sh -c foreman-rake upgrade:run 2>&1
17479 pts/3    S+     0:00 /bin/bash /sbin/foreman-rake upgrade:run
17481 pts/3    S+     0:00 runuser - foreman -s /bin/bash -c RUBYOPT=-W0 RAILS_ENV=production "$0" "$@" -- /usr/bin/rake upgrade:run
17482 ?        Rsl   83:58 /usr/bin/ruby /usr/bin/rake upgrade:run <======= IMPORTANT OUTPUT 83:58 minutes
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.