Cluster node unable to access storage after being fenced with fence_scsi and rebooting in RHEL 6

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 6 with the High Availability Add on
  • One or more nodes configured to use the fence_scsi fencing agent

Issue

  • Cluster node unable to access storage while rejoining the cluster after being fenced with fence_scsi and rebooting
  • Cluster node is getting reservation conflicts even after unfencing itself
  • When we wanted to test if fencing with fence_scsi works correctly and removed one node out of the cluster (shut down the server) the key of that node was removed from the disks (which seem to be the normal behavior). However the stopped node could never return into the cluster because it stopped during the boot process, as it could not gain access to all the devices.
  • SCSI reservation conflicts, I/O errors, and multipath map path failures occur when a node is rebooting after it has been fenced by fence_scsi
Jul  4 13:35:04 node1 kernel: sd 2:0:1:11: reservation conflict
Jul  4 13:35:29 node1 kernel: sd 2:0:1:5: reservation conflict
Jul  4 13:35:32 node1 kernel: sd 2:0:1:11: reservation conflict
Jul  4 13:35:33 node1 kernel: sd 1:0:0:5: reservation conflict
Jul  4 13:35:41 node1 kernel: sd 2:0:0:1: reservation conflict
Jul  4 13:38:52 node1 kernel: sd 2:0:1:11: reservation conflict
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Device not ready
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Sense Key : Not Ready [current] 
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Add. Sense: Logical unit not ready, manual intervention required
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] CDB: Read(10): 28 00 00 00 00 00 00 00 08 00
Jul  4 13:40:25 node1 kernel: end_request: I/O error, dev sde, sector 0
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Device not ready
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Sense Key : Not Ready [current] 
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] Add. Sense: Logical unit not ready, manual intervention required
Jul  4 13:40:25 node1 kernel: sd 1:0:0:3: [sde] CDB: Read(10): 28 00 0c 7f ff 80 00 00 08 00

Resolution

  • Ensure there is an unfencing section and device corresponding the to device in the fencing section for that node.

  • If a multipath solution is deployed on the nodes (such as device-mapper-multipath or EMC PowerPath), ensure that fence_scsi is using the appropriate device, as opposed to one of the underlying paths.

    • This is normally only an issue if devices are manually specified in the <fencedevice> or <device> tag of /etc/cluster/cluster.conf, and those specified devices point to the wrong device nodes (such as /dev/sdX or /dev/disk/by-id/scsi-XXXXXXX).
    • If no devices are specified in the fencedevice definition and fence_scsi is incorrectly reserving and registering an individual device path instead of the multipath device, then configure lvm2 to avoid scanning underlying device paths.

Root Cause

The unfencing section and device causes a node to re-activate its SCSI Persistent Reservation registration keys while starting the cman service. If this section or device is absent, the node will remain fenced from the devices, thus causing reservation conflicts, I/O errors, and/or path failures.

Even if the node is being unfenced properly during startup, if the devices are setup incorrectly in cluster.conf or through automatic detection by fence_scsi, then a node may still experience reservation conflicts and I/O errors after being unfenced. For example, if a device-mapper-multipath map is in use for a LUN, and the <fencedevice> definition in /etc/cluster/cluster.conf has a devices attribute only pointing to a single path, then a registration will only be set for that path. Now if an application writes to the corresponding multipath map, and that I/O gets sent down a different path than what was registered, a reservation conflict will occur.

If using clustered volume groups with clvmd, then the best course of action is usually to leave the devices attribute unspecified in the <fencedevice> definition, and let it handle auto-detecting the devices to register/reserve. You only need to manually specify the devices if it should use a selection other than the physical volumes for all clustered volume groups.

SBR
Product(s)
Components

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.