leapp fails with "There is not enough space on the file system hosting /var/lib/leapp directory to extract the packages"

Solution Verified - Updated

Environment

Red Hat Enterprise Linux 7
Red Hat Enterprise Linux 8
leapp

Issue

leapp upgrade fails with

============================================================
                           ERRORS
============================================================
2023-05-26 06:03:26.539380 [ERROR] Actor: dnf_upgrade_transaction
Message: There is not enough space on the file system hosting /var/lib/leapp directory to extract the packages.
 Summary:
    Hint: Please follow the instructions in the ‘Generic case’ section of the article at: link: https://access.redhat.com/solutions/5057391
============================================================
                       END OF ERRORS
============================================================

Rebooting after leapp fails in emergency mode with the same error.

Resolution

Identify if your system has any xfs partitions with ftype=0

# for i in $(findmnt -t xfs -n -o SOURCE); do xfs_info $i | grep ftype=0; done

If you have ftype=0
If you don't have ftype=0, or have no xfs partitions

No xfs partitions or only ftype=1 xfs partitions

Error is seen before rebooting

  1. Increase the amount of space on the partition holding /var/lib/leapp in order to increase the overlay size.

  2. Check /var/log/leapp-upgrade.log for the dnf failure to show which partition requires more space.

    # grep "more space needed" /var/log/leapp-upgrade.log
At least 1044MB more space needed on the / filesystem.

  3. Ensure that the specified partition has greater than, or equal to, the amount of free space in /var/lib/leapp. In the below example we can see that / has >= the amount of free space than /var which is what we want. 

    # df -h / /var/lib/leapp
Filesystem                Size  Used Avail Use% Mounted on
/dev/mapper/rhel_r8-root   27G  7.5G   20G  28% /
/dev/mapper/rhel_r8-var    10G  5.1G    5G  50% /var

See <https://access.redhat.com/solutions/5057391#generic> for more information.

If you have any xfs partitions with ftype=0

  1. Follow the steps <https://access.redhat.com/solutions/5057391#xfs> for assistance with increasing the LEAPP_OVL_SIZE

Error is seen after rebooting

If the system has failed during the reboot phase, and you are now in emergency mode, you will need to mount the system at /sysroot and verify what partition needs more space in the leapp-upgrade.log 

EXAMPLE OUTPUT
# mount /dev/mapper/root_vg-root_lv /sysroot
# grep "more space needed" /sysroot/var/log/leapp-upgrade.log
  At least 1044MB more space needed on the / filesystem.

Next, reboot back into RHEL 7 and increase the free space in the specified filesystem.

Root Cause

Leapp uses overlays to prepare the upgrade. The size of these overlays does not accurately reflect the size of each partition they are mimicking.

The Pre Reboot issues comes when the overlay is not large enough for dnf to install the packages. This overlay size always shows as the same size as whatever partition holds /var/lib/leapp. Thus if /var/lib/leapp is too small, the overlay will show as too small, even if there is enough space in the specified partition.

The Post Reboot issue comes when dnf incorrectly assume you have enough free space for the transaction, when you do not. Upon reboot, the system will find that it does not have enough space to install the packages and fail.

The Post Reboot error which is incorrectly reported can be prevented by running the below script after the leapp upgrade step and before the reboot (DISCLAIMER: this script is just an helper that is not officially supported):
This content is not included.This content is not included.https://people.redhat.com/~cbesson/leapp/eval-real-size.py

This issue is tracked by the following Jira: This content is not included.[leapp] rhel-upgrade plugin returned a non-zero exit code during the real transaction. Fix is expected for November 2023.

Diagnostic Steps

Verify the customer is using the latest version of all leapp packages.
Review the logs in /var/log/leapp/leapp-upgrade.log for space errors
Identify if this system has any xfs ftype=0 partitions or not.
Check the free space on the system. 

# rpm -qa | grep leapp
# less /var/log/leapp/leapp-upgrade.log
# for i in $(findmnt -t xfs -n -o SOURCE); do xfs_info $i | grep ftype=0; done
# df -h

If you have ftype=0
If you don't have ftype=0, or have no xfs partitions

SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.