[leapp] At least X MB more space needed on the filesystem

Solution Verified - Updated 30 Jul 2024

Environment

Red Hat Enterprise Linux 7
- leapp
- XFS

Issue

While trying to upgrade to RHEL 8.x with leapp, the following error message is printed:
```
Error Summary
-------------
Disk Requirements:
   At least 1044MB more space needed on the / filesystem.
```
Enlarging the root filesystem doesn't fix the issue.
The file system requiring more space can be different that / above.

leapp-upgrade.log file shows the following message

[ERROR] Actor: target_userspace_creator Message: Unable to install RHEL 8 userspace packages.

leapp fails with:

There is not enough space on the file system hosting /var/lib/leapp directory to extract the packages

Resolution

Update to the latest version of leapp. This issue has been resolved in version 0.16.0-1.el7_9 and 0.16.0-2.el8 as of Nov 15th, 2023.

# yum clean all && yum update leapp\*

If you pull from a local repo, Red Hat Satellite, or RHUI, ensure that your repositories contain the latest updates.

Legacy Resolution for leapp-0.15.1-1 or lower

Note The recommendation is to update to the latest version of leapp which no longer uses overlays. This resolution is left here as deprecated resolution steps for any situation where updating is not possible

There are two possible resolutions for this issue depending on system configuration

To find out which scenario applies to your issue, follow the procedure below.

Download the eval-ovl-size.py_.txt script from the "Attachments" section on this article
Upload the eval-ovl-size.py_.txt file that you downloaded to the affected system

Move the file to the correct location and make it executable

# mv eval-ovl-size.py_.txt eval-ovl-size.py
# chmod +x eval-ovl-size.py

Execute the eval-ovl-size.py script and review the recommendations provided in the output
```
# ./eval-ovl-size.py
```
If the output recommends setting a LEAPP_OVL_SIZE variable, follow the XFS ftype=0 case resolution, otherwise, follow the Generic Case resolution

Generic case

In this scenario, overlay upper directories are created in the /var mount point under /var/lib/leapp.
The error message indicates that there is not enough space on the file system hosting /var directory to extract the packages to be installed for the target release.

Enlarge the file system hosting /var/lib/leapp directory by a few Gigabytes

In case the /var directory is a dedicated file system and LVM is used, use the lvextend command as in the example below, where <vg> is the volume group and <lv> the logical volume used for /var:
```
# lvextend -r -L +2GB <vg/lv>
```
In case the /var directory is hosted on root file system, proceed similarly for the root logical volume.

Alternatively, if you have a dedicated /var file system which is not an LVM partition, you may consider removing it for the upgrade, as explained in the following article: This content is not included.This content is not included.https://access.redhat.com/solutions/4134141

Alternatively, if you cannot afford enlarging /var, you can use a temporary dedicated file system to host /var/lib/leapp, see How to set up a dedicated /var/lib/leapp for upgrade paragraph below.

Verify that all your other core os partitions (/, /usr, /var) have enough free space to hold the packages.

# grep "more space needed" /sysroot/var/log/leapp-upgrade.log
  At least 1044MB more space needed on the / filesystem.

# df -h / /var/lib/leapp
  Filesystem                Size  Used   Avail 
  /dev/mapper/rhel-root   27G  7.0G   20G         <<<< Here we see that "/" has >= free space than /var which is what we want.
  /dev/mapper/rhel-var    10G  7.5G   2.5G

Execute the leapp upgrade command again
```
# leapp upgrade
```
In case the leapp upgrade command fails again, enlarge again the /var file system.

If you believe this process is endless (i.e. you increase the size of /var by 2GB at each iteration but this never solves the issue), contact your Red Hat Support representative.

XFS ftype=0 case

XFS file systems built with ftype=0 option (typical for systems installed with RHEL 7.4 or before) require leapp to build one disk image per XFS file system.
Each disk image is by default of size 2GB, which may not be sufficient, depending on the number of packages to be applied during upgrade.
Proceed as shown below.

NOTE: the Diagnostic Steps procedure listed a number of XFS file systems built with ftype=0 option. To save storage space, you may comment out all non-standard file systems from /etc/fstab and unmount them, since leapp will not affect those file systems (e.g. file systems mounted on /opt, /apps, etc.). This however assumes that corresponding services are not running on the system for the time of the upgrade.

Adjust the environment variable LEAPP_OVL_SIZE

As explained above, internally LEAPP_OVL_SIZE is set to 2048 (2 GB), which may not be sufficient.
Typically the associated error message will be At least XXX MB more space needed on the /usr filesystem.

If XXX is < 1024, then set LEAPP_OVL_SIZE to 3072. Otherwise adjust to a larger value (computation is XXX + 2048 + some space for security).

Do this before running the appropriate leapp upgrade command from same terminal.
```
# export LEAPP_OVL_SIZE=3072
```
This will consume more space than normal in /var.
Compute the size of the required space in /var to host the disk images.
```
# expr $(findmnt -t xfs -n -o SOURCE | wc -l) \* $LEAPP_OVL_SIZE
15360
```
In the example, due to tuning LEAPP_OVL_SIZE to 3072 and having 5 XFS file systems built with ftype=0, at least 15360 MB will be required on the file system hosting /var.
Confirm that the file system hosting /var has enough space
```
# df --block-size=1M /var
Filesystem           1M-blocks  Used Available Use% Mounted on
/dev/mapper/rhel-var      8182  2255      5928  28% /var
```
In the example above, 5928 MB are available, which is clearly not enough. You hence need to enlarge /var accordingly.
If there isn't enough space in /var, enlarge it

In case the /var directory is a dedicated file system and LVM is used, use the lvextend command as in the example below, where <vg> is the volume group and <lv> the logical volume used for /var and <XXX> the size in Megabytes required (in the example above, it would be 15360 - 5928 + 1024, 1024 being some additional space used for security):
```
# lvextend -r -L +<XXX>MB <vg/lv>
```
In case the /var directory is hosted on root file system, proceed similarly for the root logical volume.

Alternatively, if you cannot afford enlarging /var, you can use a temporary dedicated file system to host /var/lib/leapp, see How to set up a dedicated /var/lib/leapp for upgrade paragraph below.
Execute the leapp upgrade command as documented in either the RHEL 7 to RHEL 8 or RHEL 8 to RHEL 9 Upgrade Guides. For example:
```
# leapp upgrade --target 9.0
```
In case the leapp upgrade command fails again, proceed again with increasing the value of LEAPP_OVL_SIZE environment variable.

If you believe this process is endless (i.e. you increase the value of LEAPP_OVL_SIZE at each iteration but this never solves the issue), contact your Red Hat Support representative.

If a dedicated /var/lib/leapp partition is desired for the upgrade please reference How to create a dedicated /var/lib/leapp partition for leapp upgrades

Root Cause

leapp versions prior to 0.16.0-1.el7_9 and 0.16.0-2.el8 used an overlay upper directory located in the /var filesystem on top of the target directory used by the upgrade process. This was visible while stracing leapp:

47534 06:50:09.273174 execve("/usr/bin/mount", ["mount", "-t", "overlay", "overlay2", "-o", "lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work", "/var/lib/leapp/scratch/mounts/root_/system_overlay"], ["LEAPP_COMMON_TOOLS=:/etc/leapp/repos.d/system_upgrade/el7toel8/tools", "LEAPP_CURRENT_ACTOR=target_userspace_creator", ...]) = 0 <0.000602>

Diagnostic Steps

There are two scenarios where the issue can happen on older versions of leapp:

either because there isn't enough space on /var
or because upgrading requires disk images to be built and these disk images are too small.

Check if there are XFS file systems on your system

# findmnt -t xfs
#

In the example above, there are no XFS file systems. Proceed directly to Resolution - Generic case.

# findmnt -t xfs
TARGET  SOURCE                FSTYPE OPTIONS
/       /dev/mapper/rhel-root xfs    rw,relatime,seclabel,attr2,inode64,noquota
├─/usr  /dev/mapper/rhel-usr  xfs    rw,relatime,seclabel,attr2,inode64,noquota
├─/boot /dev/vda1             xfs    rw,relatime,seclabel,attr2,inode64,noquota
├─/home /dev/mapper/rhel-home xfs    rw,relatime,seclabel,attr2,inode64,noquota
└─/var  /dev/mapper/rhel-var  xfs    rw,relatime,seclabel,attr2,inode64,noquota
#

In the example above, there are XFS file systems.

Check that some XFS file systems were built with ftype=0 option
```
# for dev in $(findmnt -t xfs -n -o SOURCE); do echo "$dev: $(xfs_info $dev | grep -w -o ftype=0)"; done
#
```
In the example above, the command didn't return any output, indicating that the XFS file systems were built with default ftype=1 option, proceed directly to Resolution - Generic case.
```
# for dev in $(findmnt -t xfs -n -o SOURCE); do echo "$dev: $(xfs_info $dev | grep -w -o ftype=0)"; done
/dev/mapper/rhel-root: ftype=0
/dev/mapper/rhel-usr: ftype=0
/dev/vda1: ftype=0
/dev/mapper/rhel-home: ftype=0
/dev/mapper/rhel-var: ftype=0
#
```
In the example above, all the XFS file systems were built with ftype=0 option, indicating that the system was built prior to RHEL 7.5 was out, proceed to Resolution - XFS ftype=0 case.

SBR

Anaconda

Product(s)

Red Hat Enterprise Linux

Components

anaconda

Category

Troubleshoot

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.