How to map an oracleasm path in a vmcore

Environment

• Red Hat Enterprise Linux 7

Issue

• Need to map an oracleasm path in a vmcore in order to verify that the ASM device is using a device-mapper device rather than an underlying path such as a /dev/sd(x) device.

Resolution

• In this example, we are checking whether /dev/oracleasm/disks/VOTE03 is mapped to a dm-multipath device as we hope, or if it is mapped to a /dev/sd(x) device. If the oracleasm device is mapped to a /dev/sd(x) device, it will not have proper performance and failover/redundancy capability.

• First, find the mount point of /dev/oracleasm:

crash> mount | grep -E 'oracleasm|SUPER'
     MOUNT           SUPERBLK     TYPE   DEVNAME   DIRNAME
ffffa02cf1a30780 ffffa04dc0e41000 oracleasmfs oracleasmfs /dev/oracleasm

• Track down one level to the disks directory. We can use the d_subdirs member of the dentry struct to pull the relevant dentry.

crash> struct super_block.s_root ffffa04dc0e41000
  s_root = 0xffffa0393cabb680,

crash> px &((struct dentry *)0xffffa0393cabb680)->d_subdirs
$1 = (struct list_head *) 0xffffa0393cabb720

crash> list -H 0xffffa0393cabb720 -o dentry.d_u -s dentry.d_name.name
ffffa0393cabbb00
  d_name.name = 0xffffa0393cabbb38 ".query_disk"
ffffa0393cabba40
  d_name.name = 0xffffa0393cabba78 ".check_iid"
ffffa0393cabb980
  d_name.name = 0xffffa0393cabb9b8 ".get_iid"
ffffa0393cabb8c0
  d_name.name = 0xffffa0393cabb8f8 ".query_version"
ffffa0393cabb800
  d_name.name = 0xffffa0393cabb838 "iid"
ffffa0393cabb740    <====
  d_name.name = 0xffffa0393cabb778 "disks"    <====

• Now we can go down one more sub-directory to get the VOTE03 dentry:

crash> px &((struct dentry *)0xffffa0393cabb740)->d_subdirs
$2 = (struct list_head *) 0xffffa0393cabb7e0

crash> list -H 0xffffa0393cabb7e0 -o dentry.d_u -s dentry.d_name.name
ffffa0393cb52000    <====
  d_name.name = 0xffffa0393cb52038 "VOTE03"    <====
ffffa0393c84d140
  d_name.name = 0xffffa0393c84d178 "VOTE01"
ffffa0393ca755c0
  d_name.name = 0xffffa0393ca755f8 "OCR02"
ffffa0393c8cc3c0
  d_name.name = 0xffffa0393c8cc3f8 "VOTE02"
ffffa0393c9d1d40
  d_name.name = 0xffffa0393c9d1d78 "OCR01"
ffffa038c222b380
....

• As we know this is the device we need, we can use the files command directed to its dentry to get the underlying device. Since the i_fop is set to def_blk_fops, we know the i_rdev will translate to a block device number.

crash> files -d ffffa0393cb52000
     DENTRY           INODE           SUPERBLK     TYPE PATH
ffffa0393cb52000 ffffa0393cb007a8 ffffa04dc0e41000 BLK  /dev/oracleasm/disks/VOTE03

crash> struct inode.i_fop,i_rdev ffffa0393cb007a8 -x
  i_fop = 0xffffffff92e2f7e0 <def_blk_fops>,
  i_rdev = 0x4100020,

• Translating the i_rdev to major:minor, shows the device is 65:32:

#define MINORBITS       20
#define MINORMASK       ((1U << MINORBITS) - 1)
#define MAJOR(dev)      ((unsigned int) ((dev) >> MINORBITS))
#define MINOR(dev)      ((unsigned int) ((dev) & MINORMASK))

crash> pd ((unsigned int) ((0x4100020) >> 20))
$3 = 65

crash> pd ((unsigned int) ((0x4100020) & ((1U << 20) - 1)))
$4 = 32

• From here, in general, the major and minor number could be mapped in the gendisk taken from dev -d. However, in this case, our device is a /dev/sd(x) device that has been removed from the system.

crash> dev | grep -E 'BLK|65'
BLKDEV    NAME                GENDISK      OPERATIONS      
  65      sd                  (none)     

crash> log | grep 65:32
[19495946.268572] device-mapper: multipath: Failing path 65:32.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.