Udev's block device setups which generate /dev/disk/* should link to /dev/mpath/mpath*, not the underlying /dev/sd* units.

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) including:
    • Red Hat Enterprise Linux 5 (RHEL5)
    • Red Hat Enterprise Linux 6 (RHEL6)
  • LUN's presented to the server with multiple redundant paths to that storage.
  • device-mapper-multipath managing multiple paths to SAN or iscsi LUNS.

Issue

  • Udev's block device setups which generate /dev/disk/* should link to /dev/mpath/mpath*, not the underlying /dev/sd* units.
#ls -l /dev/disk/by-uuid/

lrwxrwxrwx 1 root root  9 Jan 18 18:18 scsi-36001517db27840001458697809c0b2fd -> ../../sda
lrwxrwxrwx 1 root root 10 Jan 18 18:18 scsi-36001517db27840001458697809c0b2fd-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Jan 18 18:18 scsi-36001517db27840001458697809c0b2fd-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f00ce20000000000000005 -> ../../sdga
lrwxrwxrwx 1 root root 11 Jan 18 18:19 scsi-360050cc000f00ce20000000000000005-part1 -> ../../sdga1
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f00ce20000000000000006 -> ../../sdge
lrwxrwxrwx 1 root root 11 Jan 18 18:19 scsi-360050cc000f00ce20000000000000006-part1 -> ../../sdge1
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f00ce20000000000000007 -> ../../sdgg
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f00ce20000000000000008 -> ../../sdgi
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f00ce20000000000000009 -> ../../sdgk
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f00d020000000000000031 -> ../../sdgc
lrwxrwxrwx 1 root root 11 Jan 18 18:19 scsi-360050cc000f00d020000000000000031-part1 -> ../../sdgc1
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f01d800000000000000034 -> ../../sdfo
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f01d800000000000000035 -> ../../sdfq
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f01d800000000000000036 -> ../../sdfs
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f01d800000000000000037 -> ../../sdfu
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f01d800000000000000038 -> ../../sdfw
lrwxrwxrwx 1 root root 10 Jan 18 18:19 scsi-360050cc000f01d800000000000000039 -> ../../sdfy

Resolution

  • There is no permanent resolution to this at this time.
  • Please contact Red Hat Support if you require further assistance with this issue.
Workaround:
  • On Red Hat Enterprise Linux 5, /dev/mapper/mpath.* devices can be used which are persistent across reboots.
  • On Red Hat Enterprise Linux 6, /dev/disk/by-id/dm-name-mpath* and /dev/disk/by-id/dm-uuid-mpath-* can be used to refer to the device-mapper-multipath device for that device UUID, or the existing

Root Cause

  • Udev creates symlinks in /dev/disk/by-id/ when it detects storage devices on the system.
    • On RHEL5, it creates scsi-XXXX and wwn-XXXX devices with links to individual paths instead of links to dm-XX device-mapper devices (the dm-multipath device).
    • On RHEL6, it creates dm-name-mpath* and dm-uuid-mpath-* devices with links to dm-XX devices (dm-multipath) correctly, but it also creates scsi-XXXX and wwn-XXXX devices to individual paths as well (same as RHEL5)
  • The result is that any application that uses /dev/disk/by-id/scsi-XXXX or /dev/disk/by-id/wwn-XXXX devices will not be protected by a multipath device in the event of that path failing.

Diagnostic Steps

If the following symptoms are present on your system, then this issue may apply to you:

  • You have device-mapper-multipath devices on your system with multipath devices:

    • RHEL5 output: 
    # multipath -ll
mpath2 (3600601601bd218007449bf9805ace111) dm-3 DGC,RAID 10
[size=7.0G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
\_ round-robin 0 [prio=4][active]
 \_ 1:0:1:1 sde 8:64  [active][ready]
 \_ 1:0:3:1 sdi 8:128 [active][ready]
 \_ 2:0:1:1 sdm 8:192 [active][ready]
 \_ 2:0:3:1 sdq 65:0  [active][ready]
\_ round-robin 0 [prio=0][enabled]
 \_ 1:0:0:1 sdc 8:32  [active][ready]
 \_ 1:0:2:1 sdg 8:96  [active][ready]
 \_ 2:0:0:1 sdk 8:160 [active][ready]
 \_ 2:0:2:1 sdo 8:224 [active][ready]
    • RHEL6 output: 
    # multipath -ll
mpathb (3600601601bd21801740693b5432ae111) dm-0 DGC,RAID 10
size=3.0G features='1 queue_if_no_path' hwhandler='1 emc' wp=rw
|-+- policy='round-robin 0' prio=1 status=active
| |- 5:0:1:0 sdd 8:48 active ready  running
| `- 5:0:3:0 sdf 8:80 active ready  running
`-+- policy='round-robin 0' prio=0 status=enabled
  |- 5:0:0:0 sdc 8:32 active ready  running
  `- 5:0:2:0 sde 8:64 active ready  running

  • In /dev/disk/by-id/, you have scsi-XXXX and wwn-XXXX devices that point to individual paths rather than dm-XX devices:

    • RHEL5 output: 
    # ls -la /dev/disk/by-id/
total 0
drwxr-xr-x 2 root root 220 May 22 22:44 .
drwxr-xr-x 6 root root 120 May 22 22:04 ..
lrwxrwxrwx 1 root root   9 May 22 22:04 scsi-3600188b04c45cb000d429c2261aa2527 -> ../../sda            <-- bad - individual path
lrwxrwxrwx 1 root root  10 May 22 22:04 scsi-3600188b04c45cb000d429c2261aa2527-part1 -> ../../sda1     <-- bad - individual path
lrwxrwxrwx 1 root root  10 May 22 22:04 scsi-3600188b04c45cb000d429c2261aa2527-part2 -> ../../sda2     <-- bad - individual path
lrwxrwxrwx 1 root root   9 May 22 22:38 scsi-3600601601bd218003246ff6b05ace111 -> ../../sdb            <-- bad - individual path
lrwxrwxrwx 1 root root  10 May 22 22:44 scsi-3600601601bd218003246ff6b05ace111-part3 -> ../../sdp3     <-- bad - individual path
lrwxrwxrwx 1 root root  10 May 22 22:44 scsi-3600601601bd218003246ff6b05ace111-part8 -> ../../sdp8     <-- bad - individual path
lrwxrwxrwx 1 root root   9 May 22 22:38 scsi-3600601601bd218007449bf9805ace111 -> ../../sdm            <-- bad - individual path
    • RHEL6 output: 
    # ls -la /dev/disk/by-id/
total 0
drwxr-xr-x. 2 root root 480 May 14 18:15 .
drwxr-xr-x. 5 root root 100 May 14 14:15 ..
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-name-mpatha -> ../../dm-1           <-- good - multipath devices (dm-1)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-name-mpathap1 -> ../../dm-2         <-- good - multipath devices (dm-2)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-name-mpathap2 -> ../../dm-3         <-- good - multipath devices (dm-3)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-name-mpathb -> ../../dm-0           <-- good - multipath devices (dm-0)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-name-mpathbp1 -> ../../dm-4         <-- good - multipath devices (dm-4)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-name-myvg-rootvol -> ../../dm-5     <-- good - multipath devices (dm-5)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-uuid-mpath-3600188b04c46ca000d426f2175f806a1 -> ../../dm-1          <-- good - multipath devices (dm-1)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-uuid-mpath-3600601601bd21801740693b5432ae111 -> ../../dm-0          <-- good - multipath devices (dm-0)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-uuid-part1-mpath-3600188b04c46ca000d426f2175f806a1 -> ../../dm-2    <-- good - multipath devices (dm-2)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-uuid-part1-mpath-3600601601bd21801740693b5432ae111 -> ../../dm-4    <-- good - multipath devices (dm-4)
lrwxrwxrwx. 1 root root  10 May 14 18:15 dm-uuid-part2-mpath-3600188b04c46ca000d426f2175f806a1 -> ../../dm-3    <-- good - multipath devices (dm-3)
lrwxrwxrwx. 1 root root   9 May 14 18:15 scsi-3600188b04c46ca000d426f2175f806a1 -> ../../sda            <-- bad - individual path (sda)
lrwxrwxrwx. 1 root root  10 May 14 18:15 scsi-3600188b04c46ca000d426f2175f806a1-part1 -> ../../sda1     <-- bad - individual path (sda1)
lrwxrwxrwx. 1 root root  10 May 14 18:15 scsi-3600188b04c46ca000d426f2175f806a1-part2 -> ../../sda2     <-- bad - individual path (sda2)
lrwxrwxrwx. 1 root root   9 May 14 18:15 scsi-3600601601bd21801740693b5432ae111 -> ../../sde            <-- bad - individual path (sde)
lrwxrwxrwx. 1 root root   9 May 14 18:15 wwn-0x600188b04c46ca000d426f2175f806a1 -> ../../sda            <-- bad - individual path (sda)
lrwxrwxrwx. 1 root root  10 May 14 18:15 wwn-0x600188b04c46ca000d426f2175f806a1-part1 -> ../../sda1     <-- bad - individual path (sda1)
lrwxrwxrwx. 1 root root  10 May 14 18:15 wwn-0x600188b04c46ca000d426f2175f806a1-part2 -> ../../sda2     <-- bad - individual path (sda2)
lrwxrwxrwx. 1 root root   9 May 14 18:15 wwn-0x600601601bd21801740693b5432ae111 -> ../../sde            <-- bad - individual path (sde)
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.