Unable to set dev_loss_tmo and fast_io_fail_tmo in multipath.conf

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 6.3
  • DM-Multipath

Issue

  • After specifying the options "fast_io_fail_tmo" and "dev_loss_tmo" in multipath.conf, these values remain unchanged.

Resolution

An errata has been released for an internal This content is not included.BZ#974129 reported for the issue mentioned here. Updating the system to device-mapper-multipath-0.4.9-72.el6 or higher will resolve the issue.

With the current fix, multipath would be setting fast_io_fail_tmo value to higher than the existing dev_loss_tmo value. It first sets it to one less than the existing value. Then, if dev_loss_tmo is also being increased, it sets that and then sets fast_io_fail to the selected value.

That is, Multipath will correctly set any valid combination of fast_io_fail_tmo and dev_loss_tmo

Root Cause

Mutipath sets fast_io_fail_tmo before setting dev_loss_tmo, and sysfs doesn't allow setting fast_io_fail_tmo to equal to or higher than dev_loss_tmo

As a consequence, if user set fast_io_fail_tmo to higher than the existing dev_loss_tmo value, multipath will be unable to set it.

Diagnostic Steps

  • You can view the current values by the following commands :

  • fast_io_fail_tmo

$ for f in /sys/class/fc_remote_ports/rport-*/fast_io_fail_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
  • dev_loss_tmo
# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done

These should be over-ridden by adding entries in /etc/multipath.conf: 

defaults {
        user_friendly_names yes
        fast_io_fail_tmo <N>
        dev_loss_tmo <N>
}

[root@localhost ~]#  for f in /sys/class/fc_remote_ports/rport-*/fast_io_fail_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-1:0-0:0x5005076303ffc4d6:off
rport-1:0-1:0x5005076303ffc4d6:off
rport-2:0-0:0x5005076303ffc4d6:off
rport-2:0-1:0x5005076303ffc4d6:off
rport-5:0-0:0x100000051e36126a:off
rport-5:0-1:0x100000051e36126a:off
rport-6:0-0:0x100000051e3637ba:off
rport-6:0-1:0x100000051e3637ba:off
[root@localhost ~]#  for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-1:0-0:0x5005076303ffc4d6:8
rport-1:0-1:0x5005076303ffc4d6:8
rport-2:0-0:0x5005076303ffc4d6:8
rport-2:0-1:0x5005076303ffc4d6:8
rport-5:0-0:0x100000051e36126a:30
rport-5:0-1:0x100000051e36126a:30
rport-6:0-0:0x100000051e3637ba:30
rport-6:0-1:0x100000051e3637ba:30
  • If we strace the multipathd daemon:
#  strace -f -tt -T -v -x -o /tmp/stracing_multipathd.txt -s 4096 multipathd -v9

In the log file in /tmp you'll have something like:

/tmp/stracing_multipathd2.txt:1974  09:14:56.540154 open("/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo", O_WRONLY) = 6 <0.000048>
/tmp/stracing_multipathd2.txt-1974  09:14:56.540283 write(6, "\x33\x30\x00", 3) = -1 EINVAL (Invalid argument) <0.000024>
/tmp/stracing_multipathd2.txt-1974  09:14:56.540377 close(6)          = 0 <0.000030>
--
/tmp/stracing_multipathd3.txt:2103  09:23:44.284830 open("/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo", O_WRONLY) = 6 <0.000055>
/tmp/stracing_multipathd3.txt-2103  09:23:44.284951 write(6, "30\0", 3) = -1 EINVAL (Invalid argument) <0.000021>
/tmp/stracing_multipathd3.txt-2103  09:23:44.285030 close(6)          = 0 <0.000020>
--
/tmp/stracing_multipathd.txt:1887  09:00:00.782075 open("/sys/class/fc_remote_ports/rport-2:0-0/fast_io_fail_tmo", O_WRONLY) = 6 <0.000051>
/tmp/stracing_multipathd.txt-1887  09:00:00.782208 write(6, "\x33\x30\x00", 3) = -1 EINVAL (Invalid argument) <0.000024>
/tmp/stracing_multipathd.txt-1887  09:00:00.782301 close(6)          = 0 <0.000023>
  • Even when you have the correct order, you will still fail to set up dev_loss_tmo sometimes. This is because 
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); donerport-2:0-0:0x50060160ba601693:120
rport-2:0-1:0x200000e08b90d42d:30
rport-2:0-2:0x200000e08b8f2388:30
rport-2:0-3:0x20000000c9802436:30
rport-3:0-0:0x50060160ba601693:2147483647
rport-3:0-1:0x200000e08b8f7684:16
rport-3:0-2:0x200000e08b90cb2e:16
rport-3:0-3:0x20000000c9802437:16
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done^C
[root@1950-pe2 ~]# echo 2147483647 > /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo 
-bash: echo: write error: Invalid argument
[root@1950-pe2 ~]# echo 21 > /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo 
[root@1950-pe2 ~]# echo 2147483646 > /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo 
-bash: echo: write error: Invalid argument
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/dev_loss_tmo 
21
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/
device/            fast_io_fail_tmo   port_id            port_state         roles              subsystem/         uevent             
dev_loss_tmo       node_name          port_name          power/             scsi_target_id     supported_classes  
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/node_name 
0x200000e08b8f7684
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-1/roles 
FCP Initiator
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-3\:0-0/roles 
FCP Target
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-2\:0-0/roles 
FCP Target
[root@1950-pe2 ~]# cat /sys/class/fc_remote_ports/rport-2\:0-1/roles 
FCP Initiator
[root@1950-pe2 ~]# echo 2147483647 > /sys/class/fc_remote_ports/rport-2\:0-1/dev_loss_tmo
-bash: echo: write error: Invalid argument
[root@1950-pe2 ~]# echo 21 > /sys/class/fc_remote_ports/rport-2\:0-1/dev_loss_tmo
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-2:0-0:0x50060160ba601693:120
rport-2:0-1:0x200000e08b90d42d:21
rport-2:0-2:0x200000e08b8f2388:30
rport-2:0-3:0x20000000c9802436:30
rport-3:0-0:0x50060160ba601693:2147483647
rport-3:0-1:0x200000e08b8f7684:21
rport-3:0-2:0x200000e08b90cb2e:16
rport-3:0-3:0x20000000c9802437:16
[root@1950-pe2 ~]# echo 600 > /sys/class/fc_remote_ports/rport-2\:0-1/dev_loss_tmo
[root@1950-pe2 ~]# for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
rport-2:0-0:0x50060160ba601693:120
rport-2:0-1:0x200000e08b90d42d:600
rport-2:0-2:0x200000e08b8f2388:30
rport-2:0-3:0x20000000c9802436:30
rport-3:0-0:0x50060160ba601693:2147483647
rport-3:0-1:0x200000e08b8f7684:21
rport-3:0-2:0x200000e08b90cb2e:16
rport-3:0-3:0x20000000c9802437:16

-As we can see in drivers/scsi/scsi_transport_fc.c:

/*
 * dev_loss_tmo: the default number of seconds that the FC transport
 *   should insulate the loss of a remote port.
 *   The maximum will be capped by the value of SCSI_DEVICE_BLOCK_MAX_TIMEOUT.
 */
static unsigned int fc_dev_loss_tmo = 60;               /* seconds */
  • And this is the max value according to drivers/scsi/scsi_priv.h:
#define SCSI_DEVICE_BLOCK_MAX_TIMEOUT 600 /* units in seconds */
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.