Unable to set custom 'dev_loss_tmo' value in RHEL7

Environment

• Red Hat Enterprise Linux 7

Issue

• The value of dev_loss_tmo is set to 20 but still getting the 50 in the output of below command:

    $ less /etc/multipath.conf
    [...]
    defaults {
            user_friendly_names     yes
            fast_io_fail_tmo        10
            dev_loss_tmo            20   ###
    }
    [...]
    
    $ for f in /sys/class/fc_remote_ports/rport-*/dev_loss_tmo; do d=$(dirname $f); echo $(basename $d):$(cat $d/node_name):$(cat $f); done
    rport-7:0-2:0x50020e8020144afa:50   <---
    rport-7:0-3:0x50020e8020144af2:50   <---
    rport-8:0-2:0x50020e8020144afb:50   <---
    rport-8:0-3:0x50020e8020144af3:50   <---
  

• How is the value of dev_loss_tmo calculated in RHEL 7 ?

Resolution

• In order to allow device-mapper-multipath to set dev_loss_tmo to a low value, please ensure that no_path_retry*polling_interval<=dev_loss_tmo. For example if the desired dev_loss_tmo is 20, no_path_retry can be 4 and polling_interval 5. Also, please note that lowering polling_interval below 5 is not recommended, especially in environments with multiple LUNs and paths. As an example, the following configuration will allow setting dev_loss_tmo to 20:

    defaults {
                user_friendly_names     yes
                fast_io_fail_tmo        10
                dev_loss_tmo            20
                polling_interval        5   ###
    }
    devices {
            device {
                    vendor                  "HITACHI"
                    product                 "DF.*"
                    path_grouping_policy    multibus
                    path_checker            tur
                    features                "1 queue_if_no_path"
                    no_path_retry           4    ###
                 }
     }
  

  Note: In environments with large numbers of paths and LUNs, the polling_interval may need to be increased to avoid a load overhead caused by too frequent checker commands. In such cases and if the dev_loss_tmo cannot be equally increased, no_path_retry will need to be reduced in order to keep no_path_retry*polling_interval<=dev_loss_tmo. As an example, the settings above, could be adjusted to polling_interval 10 and no_path_retry 2.

• Then reload the multipathd service:

    $ systemctl reload multipathd
  

Root Cause

• device-mapper-multipath in RHEL7 refuses to delete the paths sooner than no_path_retry value. This is because once those paths are gone, no retries could be done on affected paths. So, with above new changes, the minimum allowed value of the dev_loss_tmo calculated as below:

    Minimum allowed 'dev_loss_tmo' by dm-multipath =  'no_path_retry' * 'polling_interval'   <----- Minimum value of the 'dev_loss_tmo'
  

Diagnostic Steps

• The following loop can reveal the rport, its WWPN, the corresponding scsi target number (if any) and the dev_loss_tmo:

    # for i in /sys/class/fc_remote_ports/rport-*; \
    do echo -e "$(basename $i)\t$(cat $i/port_name) $(cat $i/dev_loss_tmo) $(cat $i/scsi_target_id)"; \
    done
  

• Setting verbosity 3, or higher, in multipath.conf and restarting multipathd with systemctl restart multipathd will be printing in the logs messages similar to the following, indicating the mismatch between the configuration and the actual settings applied by multipathd:

    Oct 19 16:41:48 hostname multipathd: <LUN name or WWID>: dev_loss_tmo = 20 (config file default)
    ...
    Oct 19 16:41:48 hostname multipathd: <LUN name or WWID>: update dev_loss_tmo to 50
  

  The messages will be printed at the moment multipathd restarts. The verbosity setting in multipath.conf can be deleted (or set to the default 2) and multipathd can be restarted once again, right after the test, in order to avoid continuous messages in the logs that are printed when multipathd runs with verbosity 3 or higher.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.