How can I view, create, and remove SCSI persistent reservations and keys?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 5, 6, 7, or 8

Issue

  • How to create or remove SCSI Persistent Reservations and keys?
  • Want to be able to check SCSI reservations

Resolution

Warning: Removing scsi reservations without an understanding of how the application is using them can be problematic and may lead to data corruption or other unexpected behavior.

In order to work with SCSI-3 reservations at the command line you'll need to install the sg3_utils package

$ yum install sg3_utils

SCSI-3 reservations is based on use of keys. You register (add) a key to a storage device from a host and then take out a reservation against that key. There are different kinds of reservations, exclusive write for example that allows write exclusivity from one host while blocking all writes from other hosts. Other hosts not currently holding the reservation would receive a SCSI reservation conflicts status back upon attempting a write.

NOTE: /dev/ in the following notes refers to either a /dev/sd* device or a multipath

 


To verify compatibility ( SPC-3 compliant ) of a storage device use the following command:

# sg_persist -d /dev/<DEVICE> -c


To view the keys registered on a LUN use the following command:

# sg_persist --in -k -d /dev/<DEVICE>


To view the reservations currently out on a dervice use the following command:

# sg_persist --in -r -d /dev/<DEVICE>


To view more information about keys you can try following command (this may provide more information on what has created the keys):

# sg_persist --in -s -d /dev/<DEVICE>


To register a key on a LUN use the following command: (Keys are 6 or 8 digit HEX numbers. Key numbers can be arbitrary. For example 0xDEADBEEF or 0x123ABC)

# sg_persist --out --register --param-sark=<KEY> /dev/<DEVICE>

Example of registering a key with an example device and key:

 # sg_persist --out --register --param-sark=0xDEADBEEF /dev/sda


To reserve a registered device on behalf of a registered key use the following command:

# sg_persist --out --reserve --param-rk=<KEY> --prout-type=<TYPE> /dev/<DEVICE>

    The reservation types are defined in the SCSI Primary Commands specification. Here are the reservation types from the sg_persist manpage:

        1-> write exclusive
        3-> exclusive access
        5-> write exclusive - registrants only
        6-> exclusive access - registrants only
        7-> write exclusive - all registrants
        8-> exclusive access - all registrants.

Here's an example of reserving a registered device:

 # sg_persist --out --reserve --param-rk=0xDEADBEEF --prout-type=5 /dev/sda

You can release reservations and remove (unregister) keys but you have to do it in the right order and in a way that is in line with the restrictions the keys impose. For example, you cannot remove a key before releasing any/all reservations associated with those keys. You can't remove an exclusive reservation from a host unless it is the host the reservation was created from.

Please note that when attempting to remove a reservation from a multipath device, you will need to attempt the removal on ALL of the component paths until you find the one which holds the reservation. If you attempt to release the reservation through the path that didn't request the reservation, it will normally fail. That is, reservation is I_T nexus specific (I=initiator=host bus adapter, aka storage controller, T=target, remote storage port).

WARNING: Removing scsi reservations without an understanding of how the application is using them can be problematic and may lead to data corruption or other unexpected behavior.


To release a reservation against a registered key use this command:

# sg_persist --out --release --param-rk=<KEY> --prout-type=<TYPE> /dev/<DEVICE>


To unregister a key use this command:

# sg_persist --out --register --param-rk=<KEY> /dev/<DEVICE>


To clear the reservation and all registered keys use this command:

# sg_persist --out --clear --param-rk=<KEY> /dev/<DEVICE>

SCSI reservations are fairly complex. More information can be found in the man sg_persist page and in the SCSI-3 Primary Commands specification.

Diagnostic Steps

Lets walk through an example of adding and removing a registration key against a device which has multiple paths to demonstrate the difference between key visibility vs "ownership" and how to determine which I_T nexus (host HBA controller to storage port) is the current owner of the key. Only the I_T nexus that created the key/reservation (the "owner") can remove the key or reservation.

#  multipath -ll | tail -12
mpathda (3600140508c88d8babb64655bdf5cb80e) dm-14 LIO-ORG,block-98
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 8:0:0:100 sddm    71:64   active ready running
  |- 7:0:1:102 sdcq    69:224  active ready running
  |- 9:0:1:100 sdgh    131:208 active ready running
  |- 6:0:1:102 sdaw    67:0    active ready running
  |- 8:0:1:102 sdek    128:192 active ready running
  |- 7:0:5:100 sdbu    68:128  active ready running
  |- 9:0:0:102 sdfl    130:112 active ready running
  |- 6:0:0:100 sdin    135:112 active ready running
  `- 7:0:7:100 sdiw    8:256   active ready running
     I   T
     n   a
     i   r
     t   g
     i   e
     a   t
     t
     o
     r

Above is a multipath device with 4 different storage HBA on the host side, and multiple target ports on the storage side resulting in 9 different paths to the same LUN. Next a quick check to ensure no keys currently exist on the device.

# echo "-------------------------------------------------------------"
# echo "baseline: "
# echo "============================================================="
# sg_persist --in -vv -k -d /dev/mapper/mpathda
# sg_persist --in -vv -k -d /dev/sddm
# sg_persist --in -vv -k -d /dev/sdcq
# sg_persist --in -vv -k -d /dev/sdgh
# sg_persist --in -vv -k -d /dev/sdbu
# sg_persist --in -vv -k -d /dev/sdiw

output ...
-------------------------------------------------------------
baseline:
=============================================================
open /dev/mapper/mpathda with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/mapper/mpathda with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1a, there are NO registered reservation keys
open /dev/sddm with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sddm with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1a, there are NO registered reservation keys
open /dev/sdcq with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sdcq with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1a, there are NO registered reservation keys
open /dev/sdgh with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sdgh with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1a, there are NO registered reservation keys
open /dev/sdbu with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sdbu with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1a, there are NO registered reservation keys
open /dev/sdiw with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sdiw with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1a, there are NO registered reservation keys

Simply note that in spot checking both the multipath device itself and several individual paths, we are starting with no registered keys on the LUN. Now lets add a key and repeat the same commands just above:

# echo "-------------------------------------------------------------"
#echo "add key to sddm (I_T nexus 8:x:0)"
# echo "============================================================="
# sg_persist --out --register --param-sark=0x40570010 -vvv /dev/sddm
##
# echo "re-check:"
# sg_persist --in -vv -k -d /dev/mapper/mpathda
# sg_persist --in -vv -k -d /dev/sddm
# sg_persist --in -vv -k -d /dev/sdcq
# sg_persist --in -vv -k -d /dev/sdgh
# sg_persist --in -vv -k -d /dev/sdbu
# sg_persist --in -vv -k -d /dev/sdiw

output ...
-------------------------------------------------------------
add key to sddm (I_T nexus 8:x:0)
=============================================================
open /dev/sddm with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
      duration=0 ms
open /dev/sddm with flags=0x802
    Persistent Reservation Out cmd: 5f 00 00 00 00 00 00 00 18 00
    Persistent Reservation Out parameters:
      duration=1 ms

re-check:
open /dev/mapper/mpathda with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/mapper/mpathda with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1b, 1 registered reservation key follows:
    0x40570010
open /dev/sddm with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sddm with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1b, 1 registered reservation key follows:
    0x40570010
open /dev/sdcq with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sdcq with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1b, 1 registered reservation key follows:
    0x40570010
open /dev/sdgh with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sdgh  with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1b, 1 registered reservation key follows:
    0x40570010
open /dev/sdbu with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sdbu  with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1b, 1 registered reservation key follows:
    0x40570010
open /dev/sdiw with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/sdiw  with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1b, 1 registered reservation key follows:
    0x40570010

Note that although we added the key via I_T nexus 8:x:0 (sddm), since the key is on the LUN, any and all paths from this host as well as any other hosts that share this specific LUN (WWID=600140508c88d8babb64655bdf5cb80e) will be able to see the above key that was just registered. All keys and reservations are visible to all, but only the I_T nexus that created the key or reservation is allowed to remove it. Essentially the "owner" of the key or reservation is via the path that created it. The I_T nexus is I:SCSI initiator (aka HBA) and T:storage target port, so in the case of sddm which has the scsi address of 8:0:0:100, the I_T nexus is 8_0 from the perspective of the host. To show this, lets try to remove the key from a different I_T nexus.

# echo "-------------------------------------------------------------"
# echo "rem key from sdcq (should fail, wrong I_T nexus 9:x:0):"
# echo "============================================================="
# sg_persist --out --register --param-rk=0x40570010 -vvv /dev/sdcq
##
# echo "re-check:"
# sg_persist --in -vv -k -d /dev/mapper/mpathda

output ...
-------------------------------------------------------------
rem key from sdcq (should fail, wrong I_T nexus 9:x:0):
=============================================================
open /dev/sdcq with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
      duration=0 ms
open /dev/sdcq with flags=0x802
    Persistent Reservation Out cmd: 5f 00 00 00 00 00 00 00 18 00
    Persistent Reservation Out parameters:
      duration=0 ms
persistent reserve out: scsi status: Reservation Conflict
PR out: command failed
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
 00     00 00 00 00 40 57 00 10  00 00 00 00 00 00 00 00    ....@W..........
 10     00 00 00 00 00 00 00 00                             ........
re-check:
open /dev/mapper/mpathda with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/mapper/mpathda with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1b, 1 registered reservation key follows:
    0x40570010

The remove key failed and is still listed when we access the multipath device or any of its individual paths. We can try this on several other individual paths, and they will all fail except for /dev/sddm the device used to create the key. In this case we know that sddm is the "owner" of the key, but what if we didn't know? There is a method that can be used to determine which device is the key owner:

# echo "-------------------------------------------------------------"
# echo "what I_T nexus is the owner of the key?"
# echo "============================================================="
# sg_persist --in -s -vv -d /dev/mapper/mpathda
# grep -Hv "zzz" /sys/class/fc_host/host*/port_name

open /dev/mapper/mpathda with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/mapper/mpathda with flags=0x802
    Persistent Reservation In cmd: 5e 03 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1b
    Key=0x40570010
      All target ports bit clear
      Relative port address: 0x2
      not reservation holder
      Transport Id of initiator:
        FCP-2 World Wide Name:
 00     10 00 00 90 fa 16 f4 1e  

/sys/class/fc_host/host6/port_name:0x210000e08b87de9a
/sys/class/fc_host/host7/port_name:0x210100e08ba7de9a
/sys/class/fc_host/host8/port_name:0x10000090fa16f41e
/sys/class/fc_host/host9/port_name:0x10000090fa16f41f

above shows the I{nitiator} via the FCP-2 WWN field plus the T{arget} portion 
of the I_T nexus we're looking for.  Using the port_name from sysfs we can identify the scsi host as host 8.  
If the FCP-2 WWN was not listed within this host, then we'd know that the owner of the key is a different host. 
The target portion will take another couple commands to help us identify which specific device that owns the key 
as the target port address is relative.

# multipath -ll | tail -12 | grep "8:0:"

  |- 8:0:0:100 sddm    71:64   active ready running
  |- 8:0:1:102 sdek    128:192 active ready running

..so for I_T nexus 8:* we find two devices on the mpathda device:
    . 8:0:0:100, I_T nexus 8:0, sddm and
    . 8:0:1:102, I_T nexus 8:1, sdek

Next we can access the identification inquiry page for these two devices to determine which has the relative 
port address of 0x2 that was listed in the sg_persist command output above...

# echo -n "/dev/sdek: " ; sg_inq -p 0x83 -vv /dev/sdek 2>1 | grep "Relative target port" | grep -v "designator"
# echo -n "/dev/sddm: " ; sg_inq -p 0x83 -vv /dev/sddm 2>1 | grep "Relative target port" | grep -v "designator"

/dev/sdek:       Relative target port: 0x1
/dev/sddm:       Relative target port: 0x2

Using the above commands we found that the key was owned by sddm. So lets remove the key using that device.

#  echo "-------------------------------------------------------------"
# echo "rem key from sddm (should succeed, right I_T nexus 8:x:0):"
# echo "============================================================="
# sg_persist --out --register --param-rk=0x40570010 -vvv /dev/sddm
##
# echo "re-check:"
# sg_persist --in -vv -k -d /dev/mapper/mpathda

output ...
open /dev/sddm with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
      duration=0 ms
open /dev/sddm with flags=0x802
    Persistent Reservation Out cmd: 5f 00 00 00 00 00 00 00 18 00
    Persistent Reservation Out parameters:
      duration=1 ms                         ........
re-check:
open /dev/mapper/mpathda with flags=0x800
    inquiry cdb: 12 00 00 00 24 00
open /dev/mapper/mpathda with flags=0x802
    Persistent Reservation In cmd: 5e 00 00 00 00 00 00 20 00 00
  LIO-ORG   block-98          4.0
  Peripheral device type: disk
  PR generation=0x1b, there are NO registered reservation keys

Using the "owner" of the key for key removal was successful.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.