Director upgrade script stops during RHOSP upgrade when upgrading RHCS in director-deployed environment

Solution Unverified - Updated

Environment

  • Red Hat Openstack Platform (RHOSP) 17.1
  • Red Hat Ceph Storage (RHCS) 5

Issue

During the upgrade from RHOSP 16.2 to 17.1, the director upgrade script stops executing when upgrading Red Hat Ceph Storage 4 to 5 in a director-deployed Ceph Storage environment.

Resolution

Apply the following workaround to address the issue:

  1. Log in to the undercloud as the stack user with SSH.

  2. Determine the Ceph Storage orchestrator status by viewing the logs available at /home/stack/overcloud-deploy/<stack_name>/config-download/<stack_name>/cephadm/cephadm_command.log.

    • Replace <stack_name> with the stack name of the undercloud.

  3. Review the log for a task called Get the ceph orchestrator status similar to the following example:

    2023-07-12 23:19:26,936 p=463425 u=stack n=ansible | 2023-07-12 
23:19:26.935348 | 525400d7-420c-9c3f-b529-0000000001ab |       
TASK | Get the ceph orchestrator status

  4. Monitor the log. If this task does not proceed for several minutes, and no additional tasks appear in the log after it, continue with the remainder of this procedure.

  5. Log in to a Controller node with SSH.

  6. Restart the Ceph Storage orchestrator.

    sudo cephadm shell -- ceph mgr fail <controller_node_name>
    • Replace <controller_node_name> with the name of the Controller node.

  7. Return to monitoring the command logs in the undercloud. You should observe the upgrade process continuing.

Root Cause

  • Root cause is from Ceph Bug: https://bugzilla.redhat.com/show_bug.cgi?id=2223332
  • OpenStack has workarounds tracked by https://bugzilla.redhat.com/show_bug.cgi?id=2222589 and https://bugzilla.redhat.com/show_bug.cgi?id=2238930
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.