Why does multipathd daemon report failed paths?

Environment

• Red Hat Enterprise Linux 5.5
• multipath
• EMC Clariion

Issue

• Multipathd is reporting failed paths. Found following error messages in /var/log/messages file

    May  6 10:49:01 hostname kernel: sd 1:0:1:0: emc: at SP A Port 3 (owned, default SP A)
    May  6 10:49:01 hostname multipathd: dm-9: add map (uevent)
    May  6 10:49:01 hostname multipathd: dm-9: devmap already registered
    May  6 10:49:01 hostname kernel: sd 0:0:1:0: Device not ready: <6>: Current: sense key: Not Ready
    May  6 10:49:01 hostname kernel:     Add. Sense: Logical unit not ready, manual intervention required
    May  6 10:49:01 hostname kernel:
    May  6 10:49:01 hostname kernel: end_request: I/O error, dev sdl, sector 16549394
    May  6 10:49:01 hostname kernel: device-mapper: multipath: Failing path 8:176.
    May  6 10:49:01 hostname multipathd: dm-9: add map (uevent)
    May  6 10:49:01 hostname multipathd: dm-9: devmap already registered

Found following messages in dmesg :

    end_request: I/O error, dev sdx, sector 24827154
    device-mapper: multipath: Failing path 65:112.
    sd 1:0:1:0: emc: long trespass command sent
    sd 1:0:1:0: emc: at SP A Port 3 (owned, default SP A)
    sd 0:0:0:0: emc: at SP A Port 2 (owned, default SP A)
    sd 0:0:1:0: emc: long trespass command sent
    sd 0:0:1:0: emc: at SP B Port 3 (owned, default SP A)
    sd 1:0:0:0: emc: at SP B Port 2 (owned, default SP A)
    sd 0:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
    Add. Sense: Logical unit not ready, manual intervention required

Resolution

• Remove "path_grouping_policy multibus" line from active-passive devices. 

• Always use "path_grouping_policy group_by_prio" for devices for active and passive paths.

Root Cause

• The system is experiencing failover storms from a misconfiguration in /etc/multipath.conf.  For active-passive multipath devices, Linux's multipath uses priority groups to separate passive from active paths into separate path groups.  If all paths are grouped together into a single path group by use of the "multibus" grouping policy, multipath will try and use passive paths as though they were active.
• As part of path activation with an EMC Clariion, a path will be set active, which results in previously active paths becoming passive.  This results in a configuration where paths get set passive, fail, recover and become active, which then causes the previosly active paths to go through the same failure cycle.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.