When using fence_scsi with multipath devices and specifying a "devices" attribute in RHEL, reservation conflicts are logged

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux Server 6, 7, 8, 9 (with the High Availability Add On)
  • fence_scsi
  • The shared storage uses device mapper multipath (dmm)

Issue

  • During some cluster test we have seen some entries in the log file that need our attention:
Feb 11 19:43:48 node2 kernel: sd 2:0:2:6: [sdh] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Feb 11 19:43:48 node2 kernel: sd 2:0:2:6: [sdh] CDB: Write(10): 2a 00 42 87 08 00 00 04 00 00
Feb 11 19:43:48 node2 kernel: sd 2:0:2:6: reservation conflict
Feb 11 19:43:48 node2 kernel: end_request: critical nexus error, dev sdh, sector 1116147712
  • We have configured the fence_scsi to be used as fencing device for this multipathed device but we get reservation conflicts:
  <fencedevice agent="fence_scsi" name="scsi_node1" devices="/dev/disk/by-id/scsi-3600c0ff00013e0bef982e94f01000000"/>
  <fencedevice agent="fence_scsi" name="scsi_node2" devices="/dev/disk/by-id/scsi-3600c0ff00013e0bef982e94f01000000"/>

Resolution

RHEL 6 or later pacemaker based cluster


Change the `devices` attribute on the `fence_scsi` stonith device to point to the multipath device instead of the physical disk. Follow the solution outlined in the article: [Can I add or remove devices to a fence_scsi or fence_mpath device without restarting the cluster or all the resources?](/solutions/4526971)

For example, instead of having paths to the physical devices in the devices attribute you would use the path to the multipath device in the devices attribute:

  • /dev/mapper/mpatha
  • /dev/disk/by-id/scsi-3600a09803830472f632b4e44787XXXXX
  • /dev/disk/by-id/dm-uuid-mpath-3600a09803830472f632b4e44787XXXXX

RHEL 6 CMAN based cluster


Implement one of the following solutions:
  • If all shared storage devices in the cluster are physical volumes in clustered volume groups, then remove the devices attribute altogether. For example:
   <fencedevice agent="fence_scsi" name="scsi_node2"/>

fence_scsi is capable of detecting the proper devices by determining the physical volumes for any clustered volume groups. This detection will properly use the path to the multipath device, as opposed to the underlying paths.

OR

  • Change the devices attribute on the fence_scsi fencedevice in /etc/cluster/cluster.conf to point to the multipath device instead of symlink or physical disk. For example:
    <fencedevice agent="fence_scsi" name="scsi_node2" devices="/dev/mapper/mpathb, /dev/disk/by-id/scsi-360014380125989a100005000001a0000"/>
or
    <fencedevice agent="fence_scsi" name="scsi_node2" devices="/dev/disk/by-id/dm-uuid-mpath-3600XXXX, /dev/disk/by-id/dm-uuid-mpath-360014380125989a100005000001a0000"/>

OR

Remove the fence_scsi fence/stonith device from your configuration and use fence_mpath instead.

Root Cause

If specifying a value for the devices attribute of fence_scsi, and device-mapper-multipath is in use, devices must specify the path to the multipath device, not one of the underlying slaves. Alternatively, the devices attribute can be left off, allowing fence_scsi to automatically determine which devices are to be reserved/registered based on whether they are physical volumes in clustered volume groups. This detection will use the multipath device, and not underlying paths.

Otherwise all individual paths will not be registered properly, and I/O sent to individual paths may generate SCSI reservation conflicts.

For more information please refer to the 'Diagnostic Steps' section and knowledge base article Udev's block device setups which generate /dev/disk/* should link to /dev/mpath/mpath*, not the underlying /dev/sd* units..

Diagnostic Steps

Verifying the source code of fence_scsi to see how fence_scsi will register to the multipath devices.

From the do_register function, we see the following:

  • The parameters for do_register are $host_key, $node_key and the device $dev
sub do_register ($$$)
{
    my $self = (caller(0))[3];
    my ($host_key, $node_key, $dev) = @_;

    $dev = realpath ($dev);
...

* abs_path

    my $abs_path = abs_path($file);

Uses the same algorithm as getcwd(). Symbolic links and relative-path components ("." and "..") are resolved to 
return the canonical pathname, just like realpath(3).
  • But fence_scsi do only evaluate the multipath slaves when the result of $dev do start with dm at the 5th position it checks the paths 
    if (substr ($dev, 5) =~ /^dm/) {
        my @slaves = get_mpath_slaves ($dev);
        foreach (@slaves) {
            do_register ($node_key, $_);
        }
        return;
    }

With this information the fence_scsi devices are not configured correct as e.g. '/dev/disk/by-id/scsi-3600c0ff00013e0bead82e94f01000000' is a link to a /dev/sdX device.

Verifying this on a lab machine with the following multipath device configuration:

# multipath -ll
mpathb (3600508b300922b100c22b811acf50004) dm-1 HP,MSA VOLUME
size=70G features='1 queue_if_no_path' hwhandler='0' wp=rw
|-+- policy='round-robin 0' prio=50 status=active
| |- 2:0:1:1 sdc 8:32  active ready running
| `- 3:0:1:1 sdg 8:96  active ready running
`-+- policy='round-robin 0' prio=10 status=enabled
  |- 2:0:0:1 sda 8:0   active ready running
  `- 3:0:0:1 sde 8:64  active ready running

/dev/disk/by-id/scsi-3600508b300922b100c22b811acf50004 is a link to /dev/sda

# ll /dev/disk/by-id/scsi-3600508b300922b100c22b811acf50004
lrwxrwxrwx. 1 root root 9 Feb 20 16:59 /dev/disk/by-id/scsi-3600508b300922b100c22b811acf50004 -> ../../sda

With the following perl we can check what is the result of realpath

#!/usr/bin/perl

use Cwd 'realpath';
use File::Basename;
use File::Path;
use Getopt::Std;
use POSIX;


$dev='</dev/disk/by-id/XXXX-3600508b300922b100c22b811acf50004';

$dev = realpath ($dev);

printf $dev;

Running this two times with the following devices:

$dev='/dev/disk/by-id/scsi-3600508b300922b100c22b811acf50004':
# perl fence_scsi_devices.pl 
/dev/sda

$dev='/dev/disk/by-id/dm-uuid-mpath-3600508b300922b100c22b811acf50004':
# perl fence_scsi_devices.pl 
/dev/dm-1

With all this information we need to change the fence_scsi devices
from:

* /dev/disk/by-id/scsi-3600XXXX

to:

* /dev/disk/by-id/dm-uuid-mpath-3600XXXX
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.