Red Hat Upgrade Tool - rollbacks and cleanup after upgrading RHEL 6 to RHEL 7

Solution Unverified - Updated

Issue

  • How do I roll back to RHEL 6 after an unsuccessful upgrade to RHEL 7?
  • How do I remove the rollback-related artifacts after a successful upgrade to RHEL 7?

Resolution

Overview

The rollback functionality offers an easy way for customers to restore a Red Hat Enterprise Linux (RHEL) 6 system after an unsuccessful upgrade to RHEL 7 by using snapshots created by the Red Hat Upgrade Tool (RUT). This article describes how to perform the rollback and how to clean the OS after an unsuccessful upgrade to RHEL 7. This is an extension to How do I upgrade from Red Hat Enterprise Linux 6 to Red Hat Enterprise Linux 7?

Requirements and known limitations of the rollback functionality

The rollback functionality can be used only on a subset of possible setups of RHEL 6. Requirements and known limitations of the rollback functionality are as follows:

  • The only supported architecture for rollbacks is x86_64.
  • A rollback is possible only before you migrate from GRUB Legacy to GRUB 2, which is a manual step required to finish the upgrade to RHEL 7.
  • The system must be installed on LVM volume(s). It is possible to have /var and /home stored on a different LVM volume than the rest of the system.
  • To complete a successful rollback, you must create a snapshot of all system partitions (excluding /boot and swap; the /home partition depends on the user choice).
  • A rollback can be performed only on partition snapshots that were created using the Red Hat Upgrade Tool.
  • The /boot directory must be stored on a separate partition.
  • The system must have enough free space for snapshots. The required size for each snapshot is 100% of the size of the original LVM volume. If a snapshot size is insufficient, it is impossible to use the snapshot for a rollback.

Creating snapshots by using the Red Hat Upgrade Tool

This step is a prerequisite for the rollback functionality.

WARNING: The rollback procedure was tested only on snapshots of the root partition, /var and /home. Ensure that you know which partitions store data of applications installed on your system. For example, the docker storage is on the /var partition, and when upgrading the docker package, this format of the storage is upgraded as well.

WARNING: You must use the options described in this section when you perform an in-place upgrade using the redhat-upgrade-tool command. You cannot create a snapshot using the Red Hat Upgrade Tool in a process separate from the in-place upgrade.

WARNING: If your system boots using EFI, do not use the --reboot option of the redhat-upgrade-tool command and do not reboot immediately after the redhat-upgrade-tool command is executed. Follow instructions in Creating a backup for the GRUB Legacy (UEFI) before rebooting, otherwise the bootloader might be broken in case of a failed upgrade.

To select volumes that should be used for snapshots, use the following options with the redhat-upgrade-tool command:

  • --snapshot-root-lv
  • --snapshot-lv

Both options accept the following format:

<volume-name>[:<snapshot-name>[:<snapshot-size>]]

If the <snapshot-name> is not provided, the name is auto-generated.
If the <snapshot-size> is not provided, the size is the same as of the origin.
Valid size units are r|R|h|H|b|B|s|S|k|K|m|M|g|G|t|T|p|P|e|E, the same as for the lvs --units command.

The --snapshot-root-lv option can be passed only once to the Red Hat Upgrade Tool. As a result, the Red Hat Upgrade Tool creates a snapshot of the volume with the root partition and a boot entry, which is used for a rollback to RHEL 6 if needed. The --snapshot-root-lv option requires the root partition path and it is a prerequisite for the rollback functionality.

The --snapshot-lv option can be passed multiple times to the Red Hat Upgrade Tool, with each selected volume. As a result, the Red Hat Upgrade Tool creates a snapshot of each given volume. The --snapshot-lv option must be used with the --snapshot-root-lv option.

Information about snapshots created by the Red Hat Upgrade Tool is stored in the /boot/grub/snapshot.metadata file.

If you interrupt the redhat-upgrade-tool command execution before a reboot and want to change the partitions of which the snapshot should be created, you must execute the redhat-upgrade-tool command with the --clean-snapshots option first without any additional parameters:

# redhat-upgrade-tool --clean-snapshots

Example usage of --snapshot-root-lv:

  • Create a snapshot of the root partition without specifying the snapshot name and size:

    # redhat-upgrade-tool --snapshot-root-lv VolGroup/lv_root ...

    The Red Hat Upgrade Tool creates a snapshot with the same size as the origin root volume and with an auto-generated name.

  • Create a snapshot of the root partition with a specific name and with the size of the origin root volume:

    # redhat-upgrade-tool --snapshot-root-lv VolGroup/lv_root:snap_lv_root ...

  • Create a snapshot of the root partition with a specific name and size:

     # redhat-upgrade-tool --snapshot-root-lv VolGroup/lv_root:snap_lv_root:5G ...

  • Create a snapshot of the root partition with an auto-generated name and a custom snapshot size:

    # redhat-upgrade-tool --snapshot-root-lv VolGroup/lv_root::5G ...

Example usage of --snapshot-lv:

  • Create a snapshot of the /var partition with a specific name and with the size of the origin:

    # redhat-upgrade-tool --snapshot-root-lv VolGroup/lv_root --snapshot-lv VolGroup/lv_var:var_snapshot ...

Creating a backup for the GRUB Legacy (UEFI)

This step is required to provide the rollback functionality when the upgrade fails.

The GRUB Legacy binary is deleted when the RHEL 6 grub package is removed. Therefore, you must back up the binary to enable a rollback in case of an in-place upgrade failure and to ensure that the system keeps booting using GRUB Legacy.

  1. Create a backup of the GRUB Legacy binary:

    # cp /boot/efi/EFI/redhat/grub.efi /boot/efi/EFI/redhat/grub-bk.efi

  2. Copy the GRUB Legacy configuration file:

    # cp /boot/efi/EFI/redhat/grub.conf /boot/efi/EFI/redhat/grub-bk.conf

    This is necessary because GRUB Legacy requires a configuration file with a name corresponding to the binary with a .conf suffix.

  3. Create a BootXXXX EFI variable that can be used by the firmware to boot the backup GRUB Legacy binary:

    # efibootmgr -c -L 'legacy GRUB backup' -d /dev/device_name -p 1 -l '\EFI\redhat\grub-bk.efi'

    where device_name is the block device that contains the EFI System Partition (ESP) and -p specifies the partition number of the ESP.

Rollback to the original RHEL 6 state

If you need to perform a rollback to the original state of the RHEL 6 system, you have several options how to do that.

WARNING: The rollback is impossible after migration from GRUB Legacy to GRUB 2, which is a manual step required to finish the in-place upgrade to RHEL 7.

a) Restore the original RHEL 6 system from RHEL 7 after a successful upgrade

If the upgrade has finished and you have successfully booted into the RHEL 7 system but you realize some applications do not work as expected, perform the rollback by executing:

# /boot/rollback/do_rollback

When the script finishes running, the system is automatically rebooted back to the RHEL 6 system.

b) Restore the original system if an upgrade failed (BIOS)

If the upgrade failed and you are not able to boot into the RHEL 7 system, you can perform a rollback by following these steps:

  1. Boot into the RHEL 6 Snapshot entry. Note that if you created a snapshot of other partitions than root, booting lasts longer.

  2. Perform the rollback:

    # /boot/rollback/do_rollback

    When the script finishes running, the system is automatically rebooted back to the RHEL 6 system.

c) Restore the original system if an upgrade failed (UEFI)

If the upgrade fails and the procedure does not finish completely, the GRUB Legacy binary could have been deleted when the grub package was removed, and the upgrade tool may have not been able to back it up. In this scenario, the backup boot entry is used instead. Follow these steps:

  1. Boot into the RHEL 6 Snapshot entry. If you see two RHEL 6 Snapshot entries with the same timestamp, choose the second one. If you choose an incorrect entry, reboot and try again. If you created a snapshot of other than the root partition, booting lasts longer.

  2. Restore the GRUB binary and configuration file:

    # mv /boot/efi/EFI/redhat/grub-bk.efi /boot/efi/EFI/redhat/grub.efi
# mv /boot/efi/EFI/redhat/grub-bk.conf /boot/efi/EFI/redhat/grub.conf

  3. Ensure that a boot entry exists for the original GRUB Legacy binary.
    i. Use the efibootmgr utility to verify that the boot entry still exists:

    # efibootmgr -v | grep '\\EFI\\redhat\\grub.efi'

    Example output:

    Boot0001* Linux
HD(1,GPT,542e410f-cbf2-4cce-9f5d-61c4764a5d54,0x800,0x64000)/File(\EFI\redhat\grub.efi)

    ii. If no such entry exists, create a new one:

    # efibootmgr -c -d /dev/device_name -p 1 -l '\EFI\redhat\grub.efi'

    Verify that the entry has been created and it can be found by the following command:

    # efibootmgr -v | grep '\\EFI\\redhat\\grub.efi'

  4. Remove the entry created for the backup of the GRUB Legacy binary.
    i. Find the old boot entry, referring to the \EFI\redhat\grub-bk.efi file, using the efibootmgr command:

    # efibootmgr -v | grep '\\EFI\\redhat\\grub-bk.efi'

    Example output:

    Boot0004* legacy GRUB backup
HD(1,GPT,542e410f-cbf2-4cce-9f5d-61c4764a5d54,0x800,0x64000)/File(\EFI\redhat\grub-bk.efi)

    The entry number in this case is 0004.

    ii. Remove the identified boot entry:

    # efibootmgr -B -b 0004

  5. Perform the rollback:

    # /boot/rollback/do_rollback

    When the script ends, the system is automatically rebooted back to the RHEL 6 system.


Post-rollback cleanup of the RHEL 6 system

It is recommended to perform a cleanup of the RHEL 6 system after an interrupted upgrade or after performing a rollback from RHEL 7 to RHEL 6. During an in-place upgrade, the Red Hat Upgrade Tool creates artifacts needed for the upgrade and rollback, and if the user chooses to repeat the upgrade, the system should be clean before that.

To clean the RHEL 6 system after an interrupted upgrade or after performing a rollback from RHEL 7 to RHEL 6, execute:

# yum reinstall redhat-upgrade-tool
# redhat-upgrade-tool --clean

If the system boots using EFI, remove also the backed up EFI files:

# rm -f /boot/efi/EFI/redhat/*.preupg
# rm -f /boot/efi/EFI/redhat/grub-bk.efi
# rm -f /boot/efi/EFI/redhat/grub-bk.conf

Cleaning up RHEL 7 after a successful upgrade

WARNING: You should perform this operation before you migrate from GRUB Legacy to GRUB 2.

If you created snapshots for a potential rollback, the upgrade to RHEL 7 was successful, and you decide not to perform a rollback to RHEL 6, it is recommended to remove snapshots, boot entries, and backup artifacts. To perform the cleanup, run the following script:

# /boot/rollback/do_cleanup

In case of UEFI, remove the boot entry created for the backup GRUB Legacy binary:

  1. Find the old boot entry, referring to the \EFI\redhat\grub-bk.efi file, by using the efibootmgr command:

    # efibootmgr -v | grep '\\EFI\\redhat\\grub-bk.efi'

    Example output:

    Boot0004* legacy GRUB backup
HD(1,GPT,542e410f-cbf2-4cce-9f5d-61c4764a5d54,0x800,0x64000)/File(\EFI\redhat\grub-bk.efi)

    The entry number in this case is 0004.

  2. Remove the identified boot entry:

    # efibootmgr -B -b 0004

Known issues

For a list of known issues of redhat-upgrade-tool, see this Content from github.com is not included.link. Notable common common issues include:

  • Rollback with multiple kernels: After a rollback, only one kernel entry exists, even when the original RHEL 6 system has several kernels installed. It is because the tool backs up only the kernel files of the active kernel. Users often face problems with insufficient free space on the /boot partition, and for that reason, just the active bootable kernel is stored. To make the other kernels bootable again, reinstall those that are missing.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.