RHEL 7.8: XFS filesystem can not be mounted after a kernel upgrade
Environment
- Red Hat Enterprise Linux 7
- kernel-3.10.0-1127.el7 and later
- XFS
Issue
- Following an update to
kernel-3.10.0-1127.el7, a XFS file system can not be mounted. - If the system is booted into a previous version of RHEL, the XFS file system can be mounted.
- Newly created XFS file systems on RHEL 7.8 can not be mounted.
Resolution
Red Hat Enterprise Linux 7
- A resolution to creating XFS filesystems with incorrect geometry was tracked in the following Red Hat Bugzilla
- This content is not included.Bug 1835947 - mkfs.xfs can create a filesystem with bogus suint/swidth values if the kernel reports bogus BLKIOOPT and BLKIOMIN
- The bugzilla was CLOSED as WONTFIX as the offending kernel commit is planned to be removed to avoid checking for improper geometry.
Red Hat Enterprise Linux 7
-
Install
kernel-3.10.0-1127.10.1.el7(or newer) via RHBA-2020:2355, as this errata contains the following fix tracked in - RHEL 7 This content is not included.Bug 1836292 & RHEL 7.8 This content is not included.Bug 1839697revert the kernel patch "xfs: catch bad stripe alignment configurations" (BZ#1839697)
This means RHEL7 gave up to introduce the strict check to catch bad alignment because of its less risk.
Workarounds
- Boot into a kernel prior to kernel-3.10.0-1127.el7.
- Contact your storage vendor to see if there is a firmware update to address VPD pages with Minimal IO size exceeding Optimal IO size.
- Some vendors have already released Content from lenovopress.com is not included.firmware patches to address the Content from download.lenovo.com is not included.VPD page discontinuity
Root Cause
-
Invalid geometry provided by the storage vendor in VPD pages at mkfs time, resulting in an Optimal IO size that is less than the Minimum IO size
-
# mkfs.xfsuses this information when the filesytem is created, but does not validate it -
A This content is not included.commit was included in RHEL 7.8 that checks for this invalid geometry at mount time, to avoid issues with data alignment algorithms in the allocator:
[fs] xfs: catch bad stripe alignment configurations When stripe alignments are invalid, data alignment algorithms in the allocator may not work correctly. Ensure we catch superblocks with invalid stripe alignment setups at mount time. These data alignment mismatches are now detected at mount time like this: XFS (loop0): SB stripe unit sanity check failed XFS (loop0): Metadata corruption detected at xfs_sb_read_verify+0xab/0x110, xfs_sb block 0xffffffffffffffff XFS (loop0): Unmount and run xfs_repair XFS (loop0): First 128 bytes of corrupted metadata buffer: 0000000091c2de02: 58 46 53 42 00 00 10 00 00 00 00 00 00 00 10 00 XFSB............ 0000000023bff869: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00000000cdd8c893: 17 32 37 15 ff ca 46 3d 9a 17 d3 33 04 b5 f1 a2 .27...F=...3.... 000000009fd2844f: 00 00 00 00 00 00 00 04 00 00 00 00 00 00 06 d0 ................ 0000000088e9b0bb: 00 00 00 00 00 00 06 d1 00 00 00 00 00 00 06 d2 ................ 00000000ff233a20: 00 00 00 01 00 00 10 00 00 00 00 01 00 00 00 00 ................ 000000009db0ac8b: 00 00 03 60 e1 34 02 00 08 00 00 02 00 00 00 00 ...`.4.......... 00000000f7022460: 00 00 00 00 00 00 00 00 0c 09 0b 01 0c 00 00 19 ................ XFS (loop0): SB validate failed with error -117. And the mount fails. -
Invalid geometry results in a mount failure as seen here:
# mount /dev/loop0 /mnt mount: mount /dev/loop0 on /mnt failed: Structure needs cleaning [ 187.193706] XFS (loop0): SB stripe unit sanity check failed [ 187.195605] XFS (loop0): Metadata corruption detected at xfs_sb_read_verify+0x122/0x160 [xfs], xfs_sb block 0xffffffffffffffff [ 187.199084] XFS (loop0): Unmount and run xfs_repair [ 187.200622] XFS (loop0): First 128 bytes of corrupted metadata buffer: [ 187.202657] ffff8ea7fb594000: 58 46 53 42 00 00 10 00 00 00 00 03 cd 99 9c 00 XFSB............ [ 187.205343] ffff8ea7fb594010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 187.208028] ffff8ea7fb594020: 45 a8 72 5d 0c 48 46 28 be 57 00 33 f6 b4 a1 a6 E.r].HF(.W.3.... [ 187.210709] ffff8ea7fb594030: 00 00 00 01 e0 00 00 07 00 00 00 00 00 00 08 00 ................ [ 187.213378] ffff8ea7fb594040: 00 00 00 00 00 00 08 01 00 00 00 00 00 00 08 02 ................ [ 187.216053] ffff8ea7fb594050: 00 00 00 01 0f ff fe 00 00 00 00 3d 00 00 00 00 ...........=.... [ 187.218738] ffff8ea7fb594060: 00 07 f6 00 bd a5 10 00 02 00 00 08 00 00 00 00 ................ [ 187.221402] ffff8ea7fb594070: 00 00 00 00 00 00 00 00 0c 0c 09 03 1c 00 00 01 ................ [ 187.226323] XFS (loop0): SB validate failed with error -117.
Diagnostic Steps
-
If you observe a XFS filesystem fail to mount, check the
# dmesg | tailoutput for SB stripe unit sanity check failed -
If the minimum_io_size exceeds the optimal_io_size for devices with Optimal IO size greater than zero, then this allows for this behavior to occur at mkfs time.
-
These values are provide under the sysfs filesystem:
# for i in /sys/block/sd*/queue/optimal_io_size ; do echo -n "$i: " ; cat $i ; done | column -t /sys/block/sda/queue/optimal_io_size: 262144 /sys/block/sdb/queue/optimal_io_size: 262144 /sys/block/sdc/queue/optimal_io_size: 262144 # for i in /sys/block/sd*/queue/minimum_io_size ; do echo -n "$i: " ; cat $i ; done | column -t /sys/block/sda/queue/minimum_io_size: 262144 /sys/block/sdb/queue/minimum_io_size: 1048576 /sys/block/sdc/queue/minimum_io_size: 1048576 -
When mounted under an old kernel,
# xfs_info /mount/pointwill display stripe unit and stripe width; if the stripe unit exceeds the stripe width for the filesystem, it will not mount with the RHEL7.8 GA kernel. -
You can get a list of impacted filesystems by running the below commands:
# awk '$3 == "xfs"{print $2}' /proc/self/mounts | while read mount ; do echo -n "$mount " ; xfs_info $mount | awk '$0 ~ "swidth"{gsub(/.*=/,"",$2); gsub(/.*=/,"",$3); print $2,$3}' | awk '{ if ($1 > $2) print "impacted"; else print "OK"}' ; done | column -t | tee /tmp/xfs.out / OK /boot OK /mnt/ssd impacted /mnt/u01 OK # grep impacted /tmp/xfs.out /mnt/ssd impacted # xfs_info /mnt/ssd meta-data=/dev/loop1 isize=512 agcount=61, agsize=268434944 blks = sectsz=4096 attr=2, projid32bit=1 = crc=1 finobt=0 spinodes=0 data = bsize=4096 blocks=16334298112, imaxpct=1 = sunit=256 swidth=64 blks <-------- naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal bsize=4096 blocks=521728, version=2 = sectsz=4096 sunit=1 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 -
The
sunitvalue exceedsswidth. This filesystem can not be mounted on a RHEL 7.8 kernel:# uname -r ; mount /tmp/ssd_vol.meta /mnt 3.10.0-1127.el7.x86_64 mount: mount /dev/loop0 on /mnt failed: Structure needs cleaning # dmesg | tail -15 [ 61.623658] XFS (loop0): Ending clean mount [ 70.306935] XFS (loop0): Unmounting Filesystem [ 592.003052] XFS (loop0): SB stripe unit sanity check failed [ 592.008641] XFS (loop0): Metadata corruption detected at xfs_sb_read_verify+0x122/0x160 [xfs], xfs_sb block 0xffffffffffffffff [ 592.012355] XFS (loop0): Unmount and run xfs_repair [ 592.014028] XFS (loop0): First 128 bytes of corrupted metadata buffer: [ 592.015941] ffff92a7f48cb000: 58 46 53 42 00 00 10 00 00 00 00 03 cd 99 9c 00 XFSB............ [ 592.018588] ffff92a7f48cb010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ [ 592.021771] ffff92a7f48cb020: 45 a8 72 5d 0c 48 46 28 be 57 00 33 f6 b4 a1 a6 E.r].HF(.W.3.... [ 592.024307] ffff92a7f48cb030: 00 00 00 01 e0 00 00 07 00 00 00 00 00 00 08 00 ................ [ 592.026854] ffff92a7f48cb040: 00 00 00 00 00 00 08 01 00 00 00 00 00 00 08 02 ................ [ 592.029414] ffff92a7f48cb050: 00 00 00 01 0f ff fe 00 00 00 00 3d 00 00 00 00 ...........=.... [ 592.032009] ffff92a7f48cb060: 00 07 f6 00 bd a5 10 00 02 00 00 08 00 00 00 00 ................ [ 592.034632] ffff92a7f48cb070: 00 00 00 00 00 00 00 00 0c 0c 09 03 1c 00 00 01 ................ [ 592.041324] XFS (loop0): SB validate failed with error -117.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.