Failed paths to scsi devices are deleted from the system

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 6
  • Red Hat Enterprise Linux (RHEL) 7
  • Red Hat Enterprise Linux (RHEL) 8
  • Multipath capable Fibre Channel storage

Issue

  • (Some) failed paths don't appear at all in multipath -ll output
  • After a storage outage, SCSI devices have disappeared from the system

Resolution

  • This is intended behaviour. With RHEL6 and later releases, unreachable FC devices are deleted by default.

  • If behavior more like RHEL5 is desired (which does not delete devices when failed), use of the device's fast_io_fail_tmo and dev_loss_tmo timeouts can be used to emulate RHEL5 behavior. Set fast_io_fail_tmo to the desired time for I/O to wait for the port to reappear, and set dev_loss_tmo to however long the failed device should be preserved (hours, days or weeks if desired).

    • fast_io_fail_tmo and dev_loss_tmo can be configured in multipath.conf for multipath devices, or directly under the /sys filesystem.
    • fast_io_fail_tmo must be set before dev_loss_tmo can be set to large values (greater than 10 minutes).
    • fast_io_fail_tmo must be less than dev_loss_tmo - this is enforced by the kernel SCSI layer.
    • for device-mapper-multipath version 0.4.9-43 and later, dev_loss_tmo can be set to infinity.

  • For device-mapper-multipath-0.4.9-72.el6 or later, see Unable to set dev_loss_tmo and fast_io_fail_tmo in multipath.conf. This is because, all the above conditions are not applicable with this package version and multipath would be setting fast_io_fail_tmo value to higher than the existing dev_loss_tmo value. That is, Multipath will correctly set any valid combination of fast_io_fail_tmo and dev_loss_tmo.

  • The settings should be in the defaults section of /etc/multipath.conf file to apply to all non-blacklisted devices, or in the devices { device { ... } } section if different devices require different settings.

  • For example, when using device-mapper-multipath-0.4.9-43.el6 or later, the following multipath.conf settings will emulate the RHEL5 behaviour (device paths will not be deleted) :

      defaults {
          fast_io_fail_tmo  30
          dev_loss_tmo      infinity 
  }

Root Cause

  • In RHEL 6 and later releases, the unreachable Fibre Channel devices will be deleted from the system. When the system is informed a remote port is no longer reachable, it waits a period of time to see if the port reappears. If it does not reappear before the time out is reached, by default all requests to devices on the lost port will be failed and the devices deleted. If the port reappears after device deletion, the devices will be rediscovered as new SCSI devices and may (likely) have different device names.
  • The default is to fail I/O and delete the device at the same time using the dev_loss_tmo timeout value to determine how long to wait. However, another optional timeout, fast_io_fail_tmo, can be used to de-couple I/O failure and device deletion. If fast_io_fail_tmo is set, it determines how long until all I/O to the device is failed due to the port being unavailable. The device will only be deleted once the dev_loss_tmo timeout is reached. The fast_io_fail_tmo setting must always be less than dev_loss_tmo.

Diagnostic Steps

  • kernel logs will contain the message blocked FC remote port time out: removing target and saving binding when a target is lost. All devices on the target will be deleted. E.g.

      rport-5:0-2: blocked FC remote port time out: saving binding

  • In the above example, all devices on host 5, channel 0, rport 2 will be deleted. In most cases, the rport number will equal a device's target number, but this is not guaranteed.

    Reading through the kernel's drivers/scsi/scsi_transport_fc.c (and .../scsi_priv.h) the picture of how the timeouts work can become clearer.

    At the Fibre Channel transport layer (module scsi_transport_fc), there is a configurable setting dev_loss_tmo described as

      Maximum number of seconds that the FC transport should
  insulate the loss of a remote port. Once this value is
  exceeded, the scsi target is removed. Value should be
  between 1 and SCSI_DEVICE_BLOCK_MAX_TIMEOUT if
  fast_io_fail_tmo is not set.

    (scsi_transport_fc.c#line-64). SCSI_DEVICE_BLOCK_MAX_TIMEOUT is a compile time constant of 600 seconds (scsi_priv.h#line-151).

    At scsi_transport_fc.c#line-3018 we can see how fast_io_fail_tmo fits in there: when it is set, I/O to a problematic port is killed off sooner than in the normal scenario where we wait for dev_loss_tmo to kick in and remove the SCSI target for the failed port fully.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.