What is the meaning of "xfs_log_force: error -5 returned"?

Environment

• Red Hat Enterprise Linux 5 and later
• XFS filesystem

Issue

• What is the meaning of "xfs_log_force: error -5 returned"?

Resolution

After the filesystem is shutdown, the filesystem repairing tool reports the transaction log contains metadata changes that need to be replayed:

# xfs_repair -v /dev/vdb1 
Phase 1 - find and verify superblock...
        - block cache size set to 33064 entries
Phase 2 - using internal log
        - zero log...
zero_log: head block 34427 tail block 2
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_repair.  If you are unable to mount the filesystem, then use
the -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.

As the output above suggests, the first step is to remount the filesystem so the log entries can be replayed:

# mount -vvv /dev/vdb1 /mnt/
mount: /mnt does not contain SELinux labels.
       You just mounted a file system that supports labels which does not
       contain labels, onto an SELinux box. It is likely that confined
       applications will generate AVC messages and not be allowed access to
       this file system.  For more details see restorecon(8) and mount(8).
mount: /dev/vdb1 mounted on /mnt.

# dmesg
[  316.896271] XFS (vdb1): Mounting V5 Filesystem
[  316.968855] XFS (vdb1): Starting recovery (logdev: internal)
[  317.148085] XFS (vdb1): Ending recovery (logdev: internal)

The command above will recover the log entries and data loss is not expected after the filesystem gets mounted. However, since the shutdown was caused by IO errors, it's advised that the filesystem metadata is checked so that inconsistencies can be detected and fixed. In this case, it is necessary to unmount the filesystem and run the xfs_repair tool, as the following example:

# umount /mnt

# xfs_repair -v /dev/vdb1 |& tee /tmp/xfs_repair_-v_vdb1.out

Important: It's strongly recommend the xfs_repair output is redirected to a file, so it can be attached to support case for a further Root Cause Analysis.

In some cases, the log entries cannot be replayed because they are corrupted and in order to get the filesystem mounted it will require to zero out the log using xfs_repair with the -L option:

# xfs_repair -Lv /dev/vdb1 |& tee /tmp/xfs_repair_-Lv_vdb1.out

It's expected that some metadata corruptions cannot be fixed during the recovering process skipping the log and filesystem objects (files or directories) may end up being discarded or moved under lost+found/.

In cases like this, it's strongly recommended that a thorough check is performed in the mounted filesystem to determine if there are any missing directories or files, as well verifying the data integrity, so that missing and corrupted files can be restored from backups.

Root Cause

The error code -5 seen in xfs_log_force: error -5 returned is defined in /usr/include/asm-generic/errno-base.h and it's returned after an IO request fails:

#define	EIO		 5

Since the XFS relies on its transaction log to determine the filesystem integrity, the filesystem will be shut down so that it can be either replayed in an upcoming mount, or repaired.

If this issue persists after a repair, it's strongly recommended the system logs be verified for indications of IO errors to the storage device.

Diagnostic Steps

Repeating errors in the logs seen in dmesg and in /var/log/messages:

kernel:XFS (<device>): xfs_log_force: error -5 returned.
kernel:XFS (<device>): xfs_log_force: error -5 returned.
kernel:XFS (<device>): xfs_log_force: error -5 returned.
kernel:XFS (<device>): xfs_do_force_shutdown(0x1) called from line 1063 of file fs/xfs/linux-2.6/xfs_buf.c.  Return address = 0xffffffffa02c7693

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.