Leapp can fail with a MountError (OverlayFS + XFS ftype=1)

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7, 8, 9
  • In-Place Upgrades, leapp

Issue

Leapp preupgrade does not work, sometimes without any explanation (no python traceback).
In most cases, the below python exception is raised:

MountError: Failed to create mount target directory /var/lib/leapp/scratch/mounts/root_/system_overlay/XXX

Some specific symptoms have also been reported (see the diagnostic step).

Resolution

Ensure you are using the latest version of Leapp and export the below environment variable: 

# export LEAPP_OVL_IMG_FS_EXT4=1

And retry the leapp preupgrade step.

Root Cause

In some "rare" circumstances, writes fail with OverlayFS on top of XFS.

This issue was previously tracked in This content is not included.RHEL-3330 - [leapp] IPU 7>8: EINVAL returned by mkdir() on overlayfs with XFS

Diagnostic Steps

  • It can happen with XFS ftype=1. Check with xfs_info:

    # for p in $(awk '/^\// {print $1}' /proc/mounts); do echo $p: $(xfs_info $p | grep -o ftype=.); done

  • Usually, only strace'ing the behaviour confirms we hit this case. E.g.:

      9465  13:11:09.956191 mount("overlay2", "/var/lib/leapp/scratch/mounts/root_/system_overlay", "overlay", MS_MGC_VAL, "lowerdir=/,upperdir=/var/lib/leapp/scratch/mounts/root_/upper,workdir=/var/lib/leapp/scratch/mounts/root_/work") = 0 <0.000281>
9453  13:11:10.199485 mkdir("/var/lib/leapp/scratch/mounts/root_/system_overlay/var/cache/dnf", 0777) = -1 EINVAL (Invalid argument) <0.000096>

  • Be aware you can get various symptoms, but it's usually a write on an overlayfs upperdir which fails. Sometimes, write errors are not caught (e.g.: some RPMs have not been stored in the DNF cache and the issue happens after the reboot, telling the RPM has "an incorrect checksum", because it's in fact NOT there).

SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.