Device not ready errors with EMC Clariion storage but good LVM filters

Environment

• Red Hat Enterprise Linux 5.2 through 5.5
• EMC Clariion storage (or presumably other active/passive storage)
• Good LVM filters
• ALUA not used as a path checker

Issue

• Device not ready messages in the logs:

    Sep 20 17:42:25 hostname1 kernel: end_request: I/O error, dev sda, sector 0 
    Sep 20 17:42:25 hostname1 kernel: printk: 100 messages suppressed. 
    Sep 20 17:42:25 hostname1 kernel: Buffer I/O error on device sda, logical block 0 
    Sep 20 17:42:25 hostname1 kernel: sd 0:0:0:1: Device not ready: <6>: Current: sense key: Not Ready 
    Sep 20 17:42:25 hostname1 kernel:     Add. Sense: Logical unit not ready, manual intervention required 
  

Resolution

• If the LVM filters have been deemed sane and the path checker is not ALUA then you may be hitting This content is not included.RHBZ 456447 - udev vol_id probe of passive/unreadable devices causes slow bootup
• This is caused by UDEV scanning the passive paths
• Update to errata RHBA-2011-0073 or later, this may ease the effects of this issue. The errata provides udev-095-14.24.el5.
  • messages will be limited to one try per path which would limit the number of messages
  • having a single message per path would be considered expected behavior

Diagnostic Steps

• Make sure the LVM filter in lvm.conf is set to exlcude the underlying paths:

    filter = [ "a/dev/cciss/.*/", "a/dev/mapper/mpath.*/", "r/.*/" ]
  

• Make sure the path checker in multipath.conf is not ALUA:

    path_checker            emc_clariion
  

• Run systemtap script to see who/what is accessing drive(s)

  • Run: stap showio.stp -c "sleep 30" > t.log

  • File: showio.stp

      global filehandles[8192]
    
      probe ioblock.request {
          if (sector < 4) {
              printf("ioblock.bgn - device %s sector %d %s; process %s(%d)\n",
                      devname, sector, ((rw==BIO_READ)?"rd":"wr"),
                      execname(), pid() )
              print_backtrace()
          }
       }
       
       #
       # Track opened file handles...
       probe syscall.open.return {
         if ($return != -1) {
          filehandles[pid(), $return] = user_string($filename)
          printf("file opened: %s - %s(%d)\n", filehandles[pid(),$return], execname(), pid() )
         }
      }
      probe syscall.close {
        if ([pid(), $fd] in filehandles) {
          delete filehandles[pid(), $fd]
        }
      }
      probe syscall.read.return {
        if ($return <= 0) {
          if ([pid(), $fd] in filehandles) {
            printf( "syscall.read.return @%s::%d %s(%d)\n", filehandles[pid(), $fd], $return, execname(), pid() )
          }
       }
      }
    

  • Example output:

      ioblock.bgn - device sda sector 0 rd; process sysedge(10372)
      ioblock.bgn - device sdb sector 0 rd; process sysedge(10372)
      ioblock.bgn - device sdc sector 0 rd; process sysedge(10372)
      ioblock.bgn - device sdd sector 0 rd; process sysedge(10372)
      ioblock.bgn - device sde sector 0 rd; process sysedge(10372)
      ioblock.bgn - device sdk sector 0 rd; process sysedge(10372)
      ioblock.bgn - device sdl sector 0 rd; process sysedge(10372)
      ioblock.bgn - device sdm sector 0 rd; process sysedge(10372)
      ioblock.bgn - device sdn sector 0 rd; process sysedge(10372)
      ioblock.bgn - device sdo sector 0 rd; process sysedge(10372)        
    

  • Device sda is standby path device as seen in multipath output

      mpath0 (360060160b0cb1b00c8dd506ddbd0de11) dm-4 DGC,RAID 5
      size=25G][features=1 queue_if_no_path][hwhandler=1 emc][rw]
      \_ round-robin 0 [prio=2][active]
      \_ 0:0:1:0 sdf 8:80  [active][ready]
      \_ 1:0:1:0 sdp 8:240 [active][ready]
      \_ round-robin 0 [prio=0][enabled]
      \_ 0:0:0:0 sda 8:0   [active][ready]
      \_ 1:0:0:0 sdk 8:160 [active][ready]
    

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.